2024-03-29T11:07:53.783 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-29T11:07:53.783 DEBUG:teuthology.run:Teuthology command: teuthology --description rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados tasks/rados_api_tests validater/lockdep} --archive /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622690 --verbose --owner scheduled_amathuri@teuthology --name amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi -- /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622690/orig.config.yaml 2024-03-29T11:07:53.818 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-29T11:07:53.911 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622690 branch: main description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados tasks/rados_api_tests validater/lockdep} email: null first_in_suite: false job_id: '7622690' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: lockdep: true mon client directed command retry: 5 mon election default strategy: 1 ms bind msgr2: false ms inject socket failures: 5000 ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bluestore block size: 96636764160 bluestore compression algorithm: snappy bluestore compression mode: aggressive bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request mon_bind_msgr2: false sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 owner: scheduled_amathuri@teuthology priority: 150 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 8152 sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 sleep_before_teardown: 0 subset: 99/1000 suite: rados:verify suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 targets: smithi114.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBARmukIBI6GY5wbsB78CZNIyvRkpQev8UQ1peVGN117EtbV8WzHfJBDbP9V0zPW/fCCJVQ6y9x2ZLo7xriBIpQ8= smithi176.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFy1Kl6VRcmodmVDtmzeaCCslSZmfSmeyp0er6tierAsPgfbqTAsjhhvSyWV3picBPdTnnn9YJHJGAOCuGtBLNU= tasks: - install: null - ceph: null - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-26_05:45:44 tube: smithi user: amathuri verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-29T11:07:53.911 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa; will attempt to use it 2024-03-29T11:07:53.912 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks 2024-03-29T11:07:53.912 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-29T11:07:53.914 INFO:teuthology.task.internal:Checking packages... 2024-03-29T11:07:53.939 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5dcb190da270440ef141d088eb27a9fdb8afc5f2' 2024-03-29T11:07:53.939 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-29T11:07:53.939 INFO:teuthology.packaging:ref: None 2024-03-29T11:07:53.939 INFO:teuthology.packaging:tag: None 2024-03-29T11:07:53.939 INFO:teuthology.packaging:branch: main 2024-03-29T11:07:53.939 INFO:teuthology.packaging:sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:07:53.940 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-03-29T11:07:54.185 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2530.gb74786fc 2024-03-29T11:07:54.186 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-29T11:07:54.192 INFO:teuthology.task.internal:no buildpackages task found 2024-03-29T11:07:54.192 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-29T11:07:54.225 INFO:teuthology.task.internal:Saving configuration 2024-03-29T11:07:54.238 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-29T11:07:54.278 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T11:07:54.303 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi114.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622690', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 11:02:07.795198', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBARmukIBI6GY5wbsB78CZNIyvRkpQev8UQ1peVGN117EtbV8WzHfJBDbP9V0zPW/fCCJVQ6y9x2ZLo7xriBIpQ8='} 2024-03-29T11:07:54.328 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi176.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622690', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 11:02:07.796203', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFy1Kl6VRcmodmVDtmzeaCCslSZmfSmeyp0er6tierAsPgfbqTAsjhhvSyWV3picBPdTnnn9YJHJGAOCuGtBLNU='} 2024-03-29T11:07:54.328 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-29T11:07:54.334 INFO:teuthology.task.internal:roles: ubuntu@smithi114.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-03-29T11:07:54.334 INFO:teuthology.task.internal:roles: ubuntu@smithi176.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-03-29T11:07:54.334 INFO:teuthology.run_tasks:Running task console_log... 2024-03-29T11:07:54.427 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fbc6fb6d0d0>, signals=[15]) 2024-03-29T11:07:54.427 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-29T11:07:54.433 INFO:teuthology.task.internal:Opening connections... 2024-03-29T11:07:54.433 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi114.front.sepia.ceph.com 2024-03-29T11:07:54.435 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi114.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:07:54.524 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi176.front.sepia.ceph.com 2024-03-29T11:07:54.525 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:07:54.613 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-29T11:07:54.619 DEBUG:teuthology.orchestra.run.smithi114:> uname -m 2024-03-29T11:07:54.636 INFO:teuthology.orchestra.run.smithi114.stdout:x86_64 2024-03-29T11:07:54.637 DEBUG:teuthology.orchestra.run.smithi114:> cat /etc/os-release 2024-03-29T11:07:54.693 INFO:teuthology.orchestra.run.smithi114.stdout:NAME="CentOS Stream" 2024-03-29T11:07:54.693 INFO:teuthology.orchestra.run.smithi114.stdout:VERSION="9" 2024-03-29T11:07:54.693 INFO:teuthology.orchestra.run.smithi114.stdout:ID="centos" 2024-03-29T11:07:54.693 INFO:teuthology.orchestra.run.smithi114.stdout:ID_LIKE="rhel fedora" 2024-03-29T11:07:54.693 INFO:teuthology.orchestra.run.smithi114.stdout:VERSION_ID="9" 2024-03-29T11:07:54.693 INFO:teuthology.orchestra.run.smithi114.stdout:PLATFORM_ID="platform:el9" 2024-03-29T11:07:54.693 INFO:teuthology.orchestra.run.smithi114.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-29T11:07:54.693 INFO:teuthology.orchestra.run.smithi114.stdout:ANSI_COLOR="0;31" 2024-03-29T11:07:54.693 INFO:teuthology.orchestra.run.smithi114.stdout:LOGO="fedora-logo-icon" 2024-03-29T11:07:54.694 INFO:teuthology.orchestra.run.smithi114.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-29T11:07:54.694 INFO:teuthology.orchestra.run.smithi114.stdout:HOME_URL="https://centos.org/" 2024-03-29T11:07:54.694 INFO:teuthology.orchestra.run.smithi114.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-29T11:07:54.694 INFO:teuthology.orchestra.run.smithi114.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-29T11:07:54.694 INFO:teuthology.orchestra.run.smithi114.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-29T11:07:54.694 INFO:teuthology.lock.ops:Updating smithi114.front.sepia.ceph.com on lock server 2024-03-29T11:07:54.720 DEBUG:teuthology.orchestra.run.smithi176:> uname -m 2024-03-29T11:07:54.737 INFO:teuthology.orchestra.run.smithi176.stdout:x86_64 2024-03-29T11:07:54.738 DEBUG:teuthology.orchestra.run.smithi176:> cat /etc/os-release 2024-03-29T11:07:54.795 INFO:teuthology.orchestra.run.smithi176.stdout:NAME="CentOS Stream" 2024-03-29T11:07:54.795 INFO:teuthology.orchestra.run.smithi176.stdout:VERSION="9" 2024-03-29T11:07:54.795 INFO:teuthology.orchestra.run.smithi176.stdout:ID="centos" 2024-03-29T11:07:54.795 INFO:teuthology.orchestra.run.smithi176.stdout:ID_LIKE="rhel fedora" 2024-03-29T11:07:54.795 INFO:teuthology.orchestra.run.smithi176.stdout:VERSION_ID="9" 2024-03-29T11:07:54.795 INFO:teuthology.orchestra.run.smithi176.stdout:PLATFORM_ID="platform:el9" 2024-03-29T11:07:54.795 INFO:teuthology.orchestra.run.smithi176.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-29T11:07:54.796 INFO:teuthology.orchestra.run.smithi176.stdout:ANSI_COLOR="0;31" 2024-03-29T11:07:54.796 INFO:teuthology.orchestra.run.smithi176.stdout:LOGO="fedora-logo-icon" 2024-03-29T11:07:54.796 INFO:teuthology.orchestra.run.smithi176.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-29T11:07:54.796 INFO:teuthology.orchestra.run.smithi176.stdout:HOME_URL="https://centos.org/" 2024-03-29T11:07:54.796 INFO:teuthology.orchestra.run.smithi176.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-29T11:07:54.796 INFO:teuthology.orchestra.run.smithi176.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-29T11:07:54.796 INFO:teuthology.orchestra.run.smithi176.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-29T11:07:54.796 INFO:teuthology.lock.ops:Updating smithi176.front.sepia.ceph.com on lock server 2024-03-29T11:07:54.820 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-29T11:07:54.831 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-29T11:07:54.936 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-29T11:07:54.936 DEBUG:teuthology.orchestra.run.smithi114:> test '!' -e /home/ubuntu/cephtest 2024-03-29T11:07:54.938 DEBUG:teuthology.orchestra.run.smithi176:> test '!' -e /home/ubuntu/cephtest 2024-03-29T11:07:54.955 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-29T11:07:54.964 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-29T11:07:54.964 DEBUG:teuthology.orchestra.run.smithi114:> test -z $(ls -A /var/lib/ceph) 2024-03-29T11:07:54.995 DEBUG:teuthology.orchestra.run.smithi176:> test -z $(ls -A /var/lib/ceph) 2024-03-29T11:07:55.084 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-29T11:07:55.138 INFO:teuthology.run_tasks:Running task kernel... 2024-03-29T11:07:55.152 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-29T11:07:55.152 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-29T11:07:55.153 DEBUG:teuthology.orchestra.run.smithi114:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-29T11:07:55.153 DEBUG:teuthology.orchestra.run.smithi176:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-29T11:07:55.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:07:55.169 DEBUG:teuthology.orchestra.run.smithi114:> uname -r 2024-03-29T11:07:55.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:07:55.170 DEBUG:teuthology.orchestra.run.smithi176:> uname -r 2024-03-29T11:07:55.227 INFO:teuthology.orchestra.run.smithi114.stdout:5.14.0-361.el9.x86_64 2024-03-29T11:07:55.227 INFO:teuthology.task.kernel:Running kernel on smithi114: 5.14.0-361.el9.x86_64 2024-03-29T11:07:55.227 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum install -y kernel 2024-03-29T11:07:55.228 INFO:teuthology.orchestra.run.smithi176.stdout:5.14.0-361.el9.x86_64 2024-03-29T11:07:55.228 INFO:teuthology.task.kernel:Running kernel on smithi176: 5.14.0-361.el9.x86_64 2024-03-29T11:07:55.228 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum install -y kernel 2024-03-29T11:07:57.265 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS-9 - AppStream 16 MB/s | 6.7 MB 00:00 2024-03-29T11:07:57.329 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-03-29T11:07:58.882 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS-9 - BaseOS 3.9 MB/s | 2.0 MB 00:00 2024-03-29T11:07:58.946 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS-9 - BaseOS 4.0 MB/s | 2.0 MB 00:00 2024-03-29T11:07:59.853 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - BaseOS 14 MB/s | 8.0 MB 00:00 2024-03-29T11:08:02.272 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS Stream 9 - BaseOS 2.9 MB/s | 8.0 MB 00:02 2024-03-29T11:08:02.613 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - AppStream 11 MB/s | 19 MB 00:01 2024-03-29T11:08:04.541 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS Stream 9 - AppStream 17 MB/s | 19 MB 00:01 2024-03-29T11:08:08.164 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - CRB 3.0 MB/s | 6.1 MB 00:02 2024-03-29T11:08:08.940 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS Stream 9 - CRB 6.8 MB/s | 6.1 MB 00:00 2024-03-29T11:08:09.782 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - Extras packages 33 kB/s | 16 kB 00:00 2024-03-29T11:08:10.499 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS Stream 9 - Extras packages 38 kB/s | 16 kB 00:00 2024-03-29T11:08:11.634 INFO:teuthology.orchestra.run.smithi176.stdout:Extra Packages for Enterprise Linux 21 MB/s | 21 MB 00:01 2024-03-29T11:08:11.643 INFO:teuthology.orchestra.run.smithi114.stdout:Extra Packages for Enterprise Linux 12 MB/s | 21 MB 00:01 2024-03-29T11:08:16.541 INFO:teuthology.orchestra.run.smithi176.stdout:lab-extras 35 kB/s | 1.7 kB 00:00 2024-03-29T11:08:16.649 INFO:teuthology.orchestra.run.smithi114.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2024-03-29T11:08:18.798 INFO:teuthology.orchestra.run.smithi176.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-29T11:08:18.799 INFO:teuthology.orchestra.run.smithi176.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-29T11:08:18.881 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-29T11:08:18.888 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-29T11:08:18.888 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repository Size 2024-03-29T11:08:18.888 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-29T11:08:18.889 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-03-29T11:08:18.889 INFO:teuthology.orchestra.run.smithi176.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-03-29T11:08:18.889 INFO:teuthology.orchestra.run.smithi176.stdout:Installing dependencies: 2024-03-29T11:08:18.889 INFO:teuthology.orchestra.run.smithi176.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-03-29T11:08:18.889 INFO:teuthology.orchestra.run.smithi176.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-03-29T11:08:18.889 INFO:teuthology.orchestra.run.smithi176.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-03-29T11:08:18.889 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T11:08:18.889 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-03-29T11:08:18.889 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-29T11:08:18.889 INFO:teuthology.orchestra.run.smithi176.stdout:Install 4 Packages 2024-03-29T11:08:18.889 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T11:08:18.890 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 76 M 2024-03-29T11:08:18.890 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 123 M 2024-03-29T11:08:18.890 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-03-29T11:08:18.939 INFO:teuthology.orchestra.run.smithi114.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-29T11:08:18.939 INFO:teuthology.orchestra.run.smithi114.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-29T11:08:19.035 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-03-29T11:08:19.045 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-29T11:08:19.046 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repository Size 2024-03-29T11:08:19.046 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-29T11:08:19.046 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-03-29T11:08:19.046 INFO:teuthology.orchestra.run.smithi114.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-03-29T11:08:19.046 INFO:teuthology.orchestra.run.smithi114.stdout:Installing dependencies: 2024-03-29T11:08:19.046 INFO:teuthology.orchestra.run.smithi114.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-03-29T11:08:19.046 INFO:teuthology.orchestra.run.smithi114.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-03-29T11:08:19.046 INFO:teuthology.orchestra.run.smithi114.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-03-29T11:08:19.046 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T11:08:19.046 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-03-29T11:08:19.046 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-29T11:08:19.046 INFO:teuthology.orchestra.run.smithi114.stdout:Install 4 Packages 2024-03-29T11:08:19.047 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T11:08:19.047 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 76 M 2024-03-29T11:08:19.047 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 123 M 2024-03-29T11:08:19.047 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-03-29T11:08:19.126 INFO:teuthology.orchestra.run.smithi176.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 812 kB/s | 192 kB 00:00 2024-03-29T11:08:19.375 INFO:teuthology.orchestra.run.smithi114.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 585 kB/s | 192 kB 00:00 2024-03-29T11:08:21.042 INFO:teuthology.orchestra.run.smithi114.stdout:(2/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 17 MB/s | 34 MB 00:01 2024-03-29T11:08:21.676 INFO:teuthology.orchestra.run.smithi176.stdout:(2/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 12 MB/s | 34 MB 00:02 2024-03-29T11:08:21.826 INFO:teuthology.orchestra.run.smithi176.stdout:(3/4): kernel-core-5.14.0-434.el9.x86_64.rpm 5.1 MB/s | 15 MB 00:02 2024-03-29T11:08:22.200 INFO:teuthology.orchestra.run.smithi114.stdout:(3/4): kernel-core-5.14.0-434.el9.x86_64.rpm 4.8 MB/s | 15 MB 00:03 2024-03-29T11:08:22.592 INFO:teuthology.orchestra.run.smithi114.stdout:(4/4): kernel-modules-core-5.14.0-434.el9.x86_6 8.6 MB/s | 28 MB 00:03 2024-03-29T11:08:22.595 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:08:22.595 INFO:teuthology.orchestra.run.smithi114.stdout:Total 22 MB/s | 76 MB 00:03 2024-03-29T11:08:22.595 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-03-29T11:08:22.610 INFO:teuthology.orchestra.run.smithi176.stdout:(4/4): kernel-modules-core-5.14.0-434.el9.x86_6 7.9 MB/s | 28 MB 00:03 2024-03-29T11:08:22.613 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:08:22.613 INFO:teuthology.orchestra.run.smithi176.stdout:Total 21 MB/s | 76 MB 00:03 2024-03-29T11:08:22.613 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-03-29T11:08:22.975 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-03-29T11:08:22.975 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-03-29T11:08:22.976 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-03-29T11:08:22.976 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-03-29T11:08:23.800 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-03-29T11:08:23.800 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-03-29T11:08:23.821 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-03-29T11:08:23.822 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-03-29T11:08:26.128 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-03-29T11:08:26.146 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-03-29T11:08:26.707 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-03-29T11:08:26.726 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-03-29T11:08:26.792 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:08:26.817 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:08:29.539 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:08:29.593 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:08:29.676 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:08:29.735 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:08:34.348 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:08:34.480 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:08:34.491 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:08:34.517 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:08:39.223 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:08:39.257 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:09:08.223 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:09:08.223 INFO:teuthology.orchestra.run.smithi176.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-29T11:09:08.224 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T11:09:08.267 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:09:08.640 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:09:08.640 INFO:teuthology.orchestra.run.smithi114.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-29T11:09:08.641 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T11:09:08.684 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:09:08.689 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:09:08.689 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-03-29T11:09:08.690 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:09:08.690 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:09:09.067 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:09:09.067 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T11:09:09.067 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-03-29T11:09:09.067 INFO:teuthology.orchestra.run.smithi176.stdout: kernel-5.14.0-434.el9.x86_64 2024-03-29T11:09:09.067 INFO:teuthology.orchestra.run.smithi176.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-03-29T11:09:09.068 INFO:teuthology.orchestra.run.smithi176.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-03-29T11:09:09.068 INFO:teuthology.orchestra.run.smithi176.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-03-29T11:09:09.068 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T11:09:09.068 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-29T11:09:09.102 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:09:09.102 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-03-29T11:09:09.102 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:09:09.102 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:09:09.378 DEBUG:teuthology.orchestra.run.smithi176:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T11:09:09.474 INFO:teuthology.orchestra.run.smithi176.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T11:09:09.475 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T11:09:09.475 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-03-29T11:09:09.475 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-29T11:09:09.475 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-29T11:09:09.475 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-29T11:09:09.475 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi176.front.sepia.ceph.com, path=None, version=distro) 2024-03-29T11:09:09.475 DEBUG:teuthology.orchestra.run.smithi176:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T11:09:09.512 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:09:09.512 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T11:09:09.512 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-03-29T11:09:09.512 INFO:teuthology.orchestra.run.smithi114.stdout: kernel-5.14.0-434.el9.x86_64 2024-03-29T11:09:09.512 INFO:teuthology.orchestra.run.smithi114.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-03-29T11:09:09.512 INFO:teuthology.orchestra.run.smithi114.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-03-29T11:09:09.512 INFO:teuthology.orchestra.run.smithi114.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-03-29T11:09:09.512 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T11:09:09.512 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-03-29T11:09:09.555 INFO:teuthology.orchestra.run.smithi176.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T11:09:09.556 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T11:09:09.556 DEBUG:teuthology.orchestra.run.smithi176:> sudo rpm -qi grub2-tools 2024-03-29T11:09:09.589 INFO:teuthology.orchestra.run.smithi176.stdout:Name : grub2-tools 2024-03-29T11:09:09.589 INFO:teuthology.orchestra.run.smithi176.stdout:Epoch : 1 2024-03-29T11:09:09.589 INFO:teuthology.orchestra.run.smithi176.stdout:Version : 2.06 2024-03-29T11:09:09.589 INFO:teuthology.orchestra.run.smithi176.stdout:Release : 68.el9 2024-03-29T11:09:09.589 INFO:teuthology.orchestra.run.smithi176.stdout:Architecture: x86_64 2024-03-29T11:09:09.589 INFO:teuthology.orchestra.run.smithi176.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-29T11:09:09.589 INFO:teuthology.orchestra.run.smithi176.stdout:Group : Unspecified 2024-03-29T11:09:09.589 INFO:teuthology.orchestra.run.smithi176.stdout:Size : 8264254 2024-03-29T11:09:09.589 INFO:teuthology.orchestra.run.smithi176.stdout:License : GPLv3+ 2024-03-29T11:09:09.589 INFO:teuthology.orchestra.run.smithi176.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-29T11:09:09.589 INFO:teuthology.orchestra.run.smithi176.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-29T11:09:09.590 INFO:teuthology.orchestra.run.smithi176.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-29T11:09:09.590 INFO:teuthology.orchestra.run.smithi176.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-29T11:09:09.590 INFO:teuthology.orchestra.run.smithi176.stdout:Packager : builder@centos.org 2024-03-29T11:09:09.590 INFO:teuthology.orchestra.run.smithi176.stdout:Vendor : CentOS 2024-03-29T11:09:09.590 INFO:teuthology.orchestra.run.smithi176.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-29T11:09:09.590 INFO:teuthology.orchestra.run.smithi176.stdout:Summary : Support tools for GRUB. 2024-03-29T11:09:09.590 INFO:teuthology.orchestra.run.smithi176.stdout:Description : 2024-03-29T11:09:09.590 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T11:09:09.590 INFO:teuthology.orchestra.run.smithi176.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-29T11:09:09.590 INFO:teuthology.orchestra.run.smithi176.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-29T11:09:09.590 INFO:teuthology.orchestra.run.smithi176.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-29T11:09:09.590 INFO:teuthology.orchestra.run.smithi176.stdout:hardware devices. 2024-03-29T11:09:09.590 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T11:09:09.590 INFO:teuthology.orchestra.run.smithi176.stdout:This subpackage provides tools for support of all platforms. 2024-03-29T11:09:09.591 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-29T11:09:09.591 INFO:teuthology.task.kernel:Updating grub on smithi176 to boot 5.14.0-434.el9.x86_64 2024-03-29T11:09:09.591 DEBUG:teuthology.orchestra.run.smithi176:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-29T11:09:09.841 DEBUG:teuthology.orchestra.run.smithi114:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T11:09:09.943 INFO:teuthology.orchestra.run.smithi114.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T11:09:09.943 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T11:09:09.943 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-03-29T11:09:09.943 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-29T11:09:09.943 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-29T11:09:09.943 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-29T11:09:09.943 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi114.front.sepia.ceph.com, path=None, version=distro) 2024-03-29T11:09:09.944 DEBUG:teuthology.orchestra.run.smithi114:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T11:09:10.024 INFO:teuthology.orchestra.run.smithi114.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T11:09:10.024 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T11:09:10.024 DEBUG:teuthology.orchestra.run.smithi114:> sudo rpm -qi grub2-tools 2024-03-29T11:09:10.058 INFO:teuthology.orchestra.run.smithi114.stdout:Name : grub2-tools 2024-03-29T11:09:10.058 INFO:teuthology.orchestra.run.smithi114.stdout:Epoch : 1 2024-03-29T11:09:10.058 INFO:teuthology.orchestra.run.smithi114.stdout:Version : 2.06 2024-03-29T11:09:10.058 INFO:teuthology.orchestra.run.smithi114.stdout:Release : 68.el9 2024-03-29T11:09:10.058 INFO:teuthology.orchestra.run.smithi114.stdout:Architecture: x86_64 2024-03-29T11:09:10.058 INFO:teuthology.orchestra.run.smithi114.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-29T11:09:10.058 INFO:teuthology.orchestra.run.smithi114.stdout:Group : Unspecified 2024-03-29T11:09:10.058 INFO:teuthology.orchestra.run.smithi114.stdout:Size : 8264254 2024-03-29T11:09:10.058 INFO:teuthology.orchestra.run.smithi114.stdout:License : GPLv3+ 2024-03-29T11:09:10.059 INFO:teuthology.orchestra.run.smithi114.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-29T11:09:10.059 INFO:teuthology.orchestra.run.smithi114.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-29T11:09:10.059 INFO:teuthology.orchestra.run.smithi114.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-29T11:09:10.059 INFO:teuthology.orchestra.run.smithi114.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-29T11:09:10.059 INFO:teuthology.orchestra.run.smithi114.stdout:Packager : builder@centos.org 2024-03-29T11:09:10.059 INFO:teuthology.orchestra.run.smithi114.stdout:Vendor : CentOS 2024-03-29T11:09:10.059 INFO:teuthology.orchestra.run.smithi114.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-29T11:09:10.059 INFO:teuthology.orchestra.run.smithi114.stdout:Summary : Support tools for GRUB. 2024-03-29T11:09:10.059 INFO:teuthology.orchestra.run.smithi114.stdout:Description : 2024-03-29T11:09:10.059 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T11:09:10.059 INFO:teuthology.orchestra.run.smithi114.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-29T11:09:10.059 INFO:teuthology.orchestra.run.smithi114.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-29T11:09:10.059 INFO:teuthology.orchestra.run.smithi114.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-29T11:09:10.059 INFO:teuthology.orchestra.run.smithi114.stdout:hardware devices. 2024-03-29T11:09:10.059 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T11:09:10.060 INFO:teuthology.orchestra.run.smithi114.stdout:This subpackage provides tools for support of all platforms. 2024-03-29T11:09:10.060 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-29T11:09:10.060 INFO:teuthology.task.kernel:Updating grub on smithi114 to boot 5.14.0-434.el9.x86_64 2024-03-29T11:09:10.061 DEBUG:teuthology.orchestra.run.smithi114:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-29T11:09:10.183 INFO:teuthology.orchestra.run.smithi176.stderr:Generating grub configuration file ... 2024-03-29T11:09:10.652 INFO:teuthology.orchestra.run.smithi114.stderr:Generating grub configuration file ... 2024-03-29T11:09:11.537 INFO:teuthology.orchestra.run.smithi176.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-29T11:09:11.573 INFO:teuthology.orchestra.run.smithi176.stderr:done 2024-03-29T11:09:11.575 DEBUG:teuthology.orchestra.run.smithi176:> mktemp 2024-03-29T11:09:11.590 INFO:teuthology.orchestra.run.smithi176.stdout:/tmp/tmp.FYVm6wngrE 2024-03-29T11:09:11.591 DEBUG:teuthology.orchestra.run.smithi176:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.FYVm6wngrE 2024-03-29T11:09:11.659 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod 0666 /tmp/tmp.FYVm6wngrE 2024-03-29T11:09:11.795 DEBUG:teuthology.orchestra.remote:smithi176:/tmp/tmp.FYVm6wngrE is 6KB 2024-03-29T11:09:11.844 DEBUG:teuthology.orchestra.run.smithi176:> rm -fr /tmp/tmp.FYVm6wngrE 2024-03-29T11:09:11.858 DEBUG:teuthology.orchestra.run.smithi176:> sudo /bin/ls /boot/loader/entries || true 2024-03-29T11:09:11.922 INFO:teuthology.orchestra.run.smithi176.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-29T11:09:11.922 INFO:teuthology.orchestra.run.smithi176.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-29T11:09:11.922 INFO:teuthology.orchestra.run.smithi176.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-29T11:09:11.923 INFO:teuthology.orchestra.run.smithi176.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-03-29T11:09:11.924 DEBUG:teuthology.orchestra.run.smithi176:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-03-29T11:09:12.048 INFO:teuthology.orchestra.run.smithi114.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-29T11:09:12.073 DEBUG:teuthology.orchestra.run.smithi176:> sudo shutdown -r now 2024-03-29T11:09:12.083 INFO:teuthology.orchestra.run.smithi114.stderr:done 2024-03-29T11:09:12.091 DEBUG:teuthology.orchestra.run.smithi114:> mktemp 2024-03-29T11:09:12.105 INFO:teuthology.orchestra.run.smithi114.stdout:/tmp/tmp.26zMMHCEDg 2024-03-29T11:09:12.106 DEBUG:teuthology.orchestra.run.smithi114:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.26zMMHCEDg 2024-03-29T11:09:12.174 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 0666 /tmp/tmp.26zMMHCEDg 2024-03-29T11:09:12.308 DEBUG:teuthology.orchestra.remote:smithi114:/tmp/tmp.26zMMHCEDg is 6KB 2024-03-29T11:09:12.356 DEBUG:teuthology.orchestra.run.smithi114:> rm -fr /tmp/tmp.26zMMHCEDg 2024-03-29T11:09:12.370 DEBUG:teuthology.orchestra.run.smithi114:> sudo /bin/ls /boot/loader/entries || true 2024-03-29T11:09:12.435 INFO:teuthology.orchestra.run.smithi114.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-29T11:09:12.435 INFO:teuthology.orchestra.run.smithi114.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-29T11:09:12.436 INFO:teuthology.orchestra.run.smithi114.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-29T11:09:12.436 INFO:teuthology.orchestra.run.smithi114.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-03-29T11:09:12.437 DEBUG:teuthology.orchestra.run.smithi114:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-03-29T11:09:12.593 DEBUG:teuthology.orchestra.run.smithi114:> sudo shutdown -r now 2024-03-29T11:09:42.105 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-29T11:09:42.105 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-03-29T11:09:42.106 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:09:42.597 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-29T11:09:42.597 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com' 2024-03-29T11:09:42.598 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi114.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:10:00.479 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.176 2024-03-29T11:10:00.990 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.114 2024-03-29T11:10:09.485 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-03-29T11:10:09.486 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:10:09.992 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com' 2024-03-29T11:10:09.992 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi114.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:10:12.542 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.176 2024-03-29T11:10:13.054 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.114 2024-03-29T11:10:24.554 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-03-29T11:10:24.555 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:10:25.056 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com' 2024-03-29T11:10:25.057 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi114.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:10:42.974 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.176 2024-03-29T11:10:56.905 DEBUG:teuthology.orchestra.run.smithi114:> true 2024-03-29T11:10:57.347 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi114.front.sepia.ceph.com' 2024-03-29T11:10:57.347 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-434.el9.x86_64"... 2024-03-29T11:10:57.347 DEBUG:teuthology.orchestra.run.smithi114:> uname -r 2024-03-29T11:10:57.361 INFO:teuthology.orchestra.run.smithi114.stdout:5.14.0-434.el9.x86_64 2024-03-29T11:10:57.362 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-03-29T11:10:57.362 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-29T11:10:57.362 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-29T11:10:57.976 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-03-29T11:10:57.977 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:10:58.357 DEBUG:teuthology.orchestra.run.smithi176:> true 2024-03-29T11:10:58.363 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-29T11:10:58.363 DEBUG:teuthology.orchestra.run.smithi114:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-29T11:10:58.453 INFO:teuthology.orchestra.run.smithi114.stdout:ttyS1 2024-03-29T11:10:58.496 DEBUG:teuthology.parallel:result is None 2024-03-29T11:10:58.782 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-03-29T11:10:58.782 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-434.el9.x86_64"... 2024-03-29T11:10:58.783 DEBUG:teuthology.orchestra.run.smithi176:> uname -r 2024-03-29T11:10:58.797 INFO:teuthology.orchestra.run.smithi176.stdout:5.14.0-434.el9.x86_64 2024-03-29T11:10:58.797 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-03-29T11:10:58.797 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-29T11:10:58.797 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-29T11:10:59.799 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-29T11:10:59.799 DEBUG:teuthology.orchestra.run.smithi176:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-29T11:10:59.877 INFO:teuthology.orchestra.run.smithi176.stdout:ttyS1 2024-03-29T11:10:59.912 DEBUG:teuthology.parallel:result is None 2024-03-29T11:10:59.913 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-29T11:10:59.922 INFO:teuthology.task.internal:Creating test directory... 2024-03-29T11:10:59.923 DEBUG:teuthology.orchestra.run.smithi114:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-29T11:10:59.925 DEBUG:teuthology.orchestra.run.smithi176:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-29T11:10:59.939 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-29T11:10:59.967 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-29T11:11:00.003 INFO:teuthology.task.internal:Creating archive directory... 2024-03-29T11:11:00.004 DEBUG:teuthology.orchestra.run.smithi114:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-29T11:11:00.006 DEBUG:teuthology.orchestra.run.smithi176:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-29T11:11:00.031 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-29T11:11:00.040 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-29T11:11:00.040 DEBUG:teuthology.orchestra.run.smithi114:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-29T11:11:00.066 DEBUG:teuthology.orchestra.run.smithi176:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-29T11:11:00.099 INFO:teuthology.orchestra.run.smithi114.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:11:00.099 INFO:teuthology.orchestra.run.smithi176.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:11:00.112 INFO:teuthology.orchestra.run.smithi114.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:11:00.113 INFO:teuthology.orchestra.run.smithi176.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:11:00.114 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-29T11:11:00.123 INFO:teuthology.task.internal:Configuring sudo... 2024-03-29T11:11:00.123 DEBUG:teuthology.orchestra.run.smithi114:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-29T11:11:00.156 DEBUG:teuthology.orchestra.run.smithi176:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-29T11:11:00.185 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-29T11:11:00.195 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-29T11:11:00.195 DEBUG:teuthology.orchestra.run.smithi114:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-29T11:11:00.223 DEBUG:teuthology.orchestra.run.smithi176:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-29T11:11:00.247 DEBUG:teuthology.orchestra.run.smithi114:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:11:00.320 DEBUG:teuthology.orchestra.run.smithi114:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:11:00.374 DEBUG:teuthology.orchestra.run.smithi114:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:11:00.456 DEBUG:teuthology.orchestra.run.smithi114:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:11:00.499 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-29T11:11:00.500 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-29T11:11:00.567 DEBUG:teuthology.orchestra.run.smithi176:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:11:00.609 DEBUG:teuthology.orchestra.run.smithi176:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:11:00.668 DEBUG:teuthology.orchestra.run.smithi176:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:11:00.759 DEBUG:teuthology.orchestra.run.smithi176:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:11:00.823 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-03-29T11:11:00.824 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-29T11:11:00.915 DEBUG:teuthology.orchestra.run.smithi114:> sudo service rsyslog restart 2024-03-29T11:11:00.918 DEBUG:teuthology.orchestra.run.smithi176:> sudo service rsyslog restart 2024-03-29T11:11:00.956 INFO:teuthology.orchestra.run.smithi114.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:11:01.049 INFO:teuthology.orchestra.run.smithi176.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:11:01.598 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-29T11:11:01.606 INFO:teuthology.task.internal:Starting timer... 2024-03-29T11:11:01.606 INFO:teuthology.run_tasks:Running task pcp... 2024-03-29T11:11:01.645 INFO:teuthology.run_tasks:Running task selinux... 2024-03-29T11:11:01.689 DEBUG:teuthology.orchestra.run.smithi114:> sudo service auditd rotate 2024-03-29T11:11:01.745 INFO:teuthology.orchestra.run.smithi114.stdout:Rotating logs: 2024-03-29T11:11:01.747 DEBUG:teuthology.orchestra.run.smithi176:> sudo service auditd rotate 2024-03-29T11:11:01.804 INFO:teuthology.orchestra.run.smithi176.stdout:Rotating logs: 2024-03-29T11:11:01.806 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-29T11:11:01.806 DEBUG:teuthology.orchestra.run.smithi114:> /usr/sbin/getenforce 2024-03-29T11:11:01.840 INFO:teuthology.orchestra.run.smithi114.stdout:Permissive 2024-03-29T11:11:01.841 DEBUG:teuthology.orchestra.run.smithi176:> /usr/sbin/getenforce 2024-03-29T11:11:01.868 INFO:teuthology.orchestra.run.smithi176.stdout:Permissive 2024-03-29T11:11:01.868 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi114.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi176.front.sepia.ceph.com': 'permissive'} 2024-03-29T11:11:01.868 DEBUG:teuthology.orchestra.run.smithi114:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:11:01.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:11:01.909 DEBUG:teuthology.orchestra.run.smithi176:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:11:01.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:11:01.936 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-29T11:11:01.937 DEBUG:teuthology.orchestra.run.smithi114:> sudo /usr/sbin/setenforce permissive 2024-03-29T11:11:01.984 DEBUG:teuthology.orchestra.run.smithi176:> sudo /usr/sbin/setenforce permissive 2024-03-29T11:11:02.120 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-29T11:11:02.130 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-29T11:11:02.146 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-29T11:11:02.171 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi114.front.sepia.ceph.com,smithi176.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-29T11:15:29.433 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi114.front.sepia.ceph.com'), Remote(name='ubuntu@smithi176.front.sepia.ceph.com')] 2024-03-29T11:15:29.434 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com' 2024-03-29T11:15:29.435 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi114.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:15:29.526 DEBUG:teuthology.orchestra.run.smithi114:> true 2024-03-29T11:15:29.609 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi114.front.sepia.ceph.com' 2024-03-29T11:15:29.610 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-03-29T11:15:29.610 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:15:29.702 DEBUG:teuthology.orchestra.run.smithi176:> true 2024-03-29T11:15:29.781 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-03-29T11:15:29.781 INFO:teuthology.run_tasks:Running task clock... 2024-03-29T11:15:29.792 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-29T11:15:29.793 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-29T11:15:29.793 DEBUG:teuthology.orchestra.run.smithi114:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:15:29.795 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-29T11:15:29.796 DEBUG:teuthology.orchestra.run.smithi176:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:15:29.825 INFO:teuthology.orchestra.run.smithi114.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-29T11:15:29.841 INFO:teuthology.orchestra.run.smithi114.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-29T11:15:29.853 INFO:teuthology.orchestra.run.smithi176.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-29T11:15:29.869 INFO:teuthology.orchestra.run.smithi176.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-29T11:15:29.872 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: ntpd: command not found 2024-03-29T11:15:29.885 INFO:teuthology.orchestra.run.smithi114.stdout:506 Cannot talk to daemon 2024-03-29T11:15:29.898 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: ntpd: command not found 2024-03-29T11:15:29.902 INFO:teuthology.orchestra.run.smithi114.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-29T11:15:29.912 INFO:teuthology.orchestra.run.smithi176.stdout:506 Cannot talk to daemon 2024-03-29T11:15:29.918 INFO:teuthology.orchestra.run.smithi114.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-29T11:15:29.928 INFO:teuthology.orchestra.run.smithi176.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-29T11:15:29.945 INFO:teuthology.orchestra.run.smithi176.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-29T11:15:29.974 INFO:teuthology.orchestra.run.smithi114.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:15:29.977 INFO:teuthology.orchestra.run.smithi114.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:15:29.977 INFO:teuthology.orchestra.run.smithi114.stdout:=============================================================================== 2024-03-29T11:15:29.977 INFO:teuthology.orchestra.run.smithi114.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:15:29.977 INFO:teuthology.orchestra.run.smithi114.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:15:29.977 INFO:teuthology.orchestra.run.smithi114.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:15:29.977 INFO:teuthology.orchestra.run.smithi114.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:15:29.999 INFO:teuthology.orchestra.run.smithi176.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:15:30.002 INFO:teuthology.orchestra.run.smithi176.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:15:30.002 INFO:teuthology.orchestra.run.smithi176.stdout:=============================================================================== 2024-03-29T11:15:30.002 INFO:teuthology.orchestra.run.smithi176.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:15:30.002 INFO:teuthology.orchestra.run.smithi176.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:15:30.002 INFO:teuthology.orchestra.run.smithi176.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:15:30.002 INFO:teuthology.orchestra.run.smithi176.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:15:30.003 INFO:teuthology.run_tasks:Running task install... 2024-03-29T11:15:30.011 DEBUG:teuthology.task.install:project ceph 2024-03-29T11:15:30.011 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}} 2024-03-29T11:15:30.011 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'} 2024-03-29T11:15:30.012 INFO:teuthology.task.install:Using flavor: default 2024-03-29T11:15:30.017 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-29T11:15:30.017 INFO:teuthology.task.install:extra packages: [] 2024-03-29T11:15:30.018 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-29T11:15:30.018 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:15:30.020 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-29T11:15:30.020 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:15:30.166 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-29T11:15:30.166 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-29T11:15:30.237 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-29T11:15:30.237 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-29T11:15:30.335 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-29T11:15:30.336 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-03-29T11:15:30.336 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-29T11:15:30.364 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-29T11:15:30.364 DEBUG:teuthology.orchestra.run.smithi176:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-29T11:15:30.432 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-29T11:15:30.433 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-29T11:15:30.433 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-29T11:15:30.434 DEBUG:teuthology.orchestra.run.smithi176:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-29T11:15:30.463 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-29T11:15:30.463 DEBUG:teuthology.orchestra.run.smithi114:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-29T11:15:30.514 DEBUG:teuthology.orchestra.run.smithi176:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-29T11:15:30.532 DEBUG:teuthology.orchestra.run.smithi114:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-29T11:15:30.581 INFO:teuthology.orchestra.run.smithi176.stdout:check_obsoletes = 1 2024-03-29T11:15:30.582 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum clean all 2024-03-29T11:15:30.613 DEBUG:teuthology.orchestra.run.smithi114:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-29T11:15:30.680 INFO:teuthology.orchestra.run.smithi114.stdout:check_obsoletes = 1 2024-03-29T11:15:30.682 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum clean all 2024-03-29T11:15:30.827 INFO:teuthology.orchestra.run.smithi176.stdout:74 files removed 2024-03-29T11:15:30.854 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph-radosgw 2024-03-29T11:15:30.937 INFO:teuthology.orchestra.run.smithi114.stdout:74 files removed 2024-03-29T11:15:30.964 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph-radosgw 2024-03-29T11:15:31.569 INFO:teuthology.orchestra.run.smithi176.stdout:ceph packages for x86_64 198 kB/s | 82 kB 00:00 2024-03-29T11:15:31.684 INFO:teuthology.orchestra.run.smithi114.stdout:ceph packages for x86_64 195 kB/s | 82 kB 00:00 2024-03-29T11:15:31.995 INFO:teuthology.orchestra.run.smithi176.stdout:ceph noarch packages 27 kB/s | 10 kB 00:00 2024-03-29T11:15:32.026 INFO:teuthology.orchestra.run.smithi114.stdout:ceph noarch packages 34 kB/s | 10 kB 00:00 2024-03-29T11:15:32.304 INFO:teuthology.orchestra.run.smithi176.stdout:ceph source packages 7.1 kB/s | 1.9 kB 00:00 2024-03-29T11:15:32.435 INFO:teuthology.orchestra.run.smithi114.stdout:ceph source packages 5.1 kB/s | 1.9 kB 00:00 2024-03-29T11:15:32.723 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS-9 - AppStream 18 MB/s | 6.7 MB 00:00 2024-03-29T11:15:32.838 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS-9 - AppStream 18 MB/s | 6.7 MB 00:00 2024-03-29T11:15:34.123 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS-9 - BaseOS 7.1 MB/s | 2.0 MB 00:00 2024-03-29T11:15:34.247 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS-9 - BaseOS 6.8 MB/s | 2.0 MB 00:00 2024-03-29T11:15:35.037 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - BaseOS 17 MB/s | 8.0 MB 00:00 2024-03-29T11:15:35.423 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS Stream 9 - BaseOS 8.2 MB/s | 8.0 MB 00:00 2024-03-29T11:15:37.681 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS Stream 9 - AppStream 16 MB/s | 19 MB 00:01 2024-03-29T11:15:42.007 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS Stream 9 - CRB 7.4 MB/s | 6.1 MB 00:00 2024-03-29T11:15:43.625 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS Stream 9 - Extras packages 34 kB/s | 16 kB 00:00 2024-03-29T11:15:44.634 INFO:teuthology.orchestra.run.smithi176.stdout:Extra Packages for Enterprise Linux 23 MB/s | 21 MB 00:00 2024-03-29T11:15:49.584 INFO:teuthology.orchestra.run.smithi176.stdout:lab-extras 26 kB/s | 1.7 kB 00:00 2024-03-29T11:15:52.197 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-29T11:15:52.205 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-29T11:15:52.205 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repository Size 2024-03-29T11:15:52.205 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-29T11:15:52.205 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-03-29T11:15:52.205 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-29T11:15:52.205 INFO:teuthology.orchestra.run.smithi176.stdout:Upgrading: 2024-03-29T11:15:52.205 INFO:teuthology.orchestra.run.smithi176.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-29T11:15:52.205 INFO:teuthology.orchestra.run.smithi176.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:15:52.205 INFO:teuthology.orchestra.run.smithi176.stdout:Installing dependencies: 2024-03-29T11:15:52.205 INFO:teuthology.orchestra.run.smithi176.stdout: boost-program-options 2024-03-29T11:15:52.206 INFO:teuthology.orchestra.run.smithi176.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-29T11:15:52.206 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-29T11:15:52.206 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-29T11:15:52.206 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-29T11:15:52.206 INFO:teuthology.orchestra.run.smithi176.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-29T11:15:52.206 INFO:teuthology.orchestra.run.smithi176.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-29T11:15:52.206 INFO:teuthology.orchestra.run.smithi176.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-29T11:15:52.206 INFO:teuthology.orchestra.run.smithi176.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-29T11:15:52.206 INFO:teuthology.orchestra.run.smithi176.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-29T11:15:52.206 INFO:teuthology.orchestra.run.smithi176.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-29T11:15:52.206 INFO:teuthology.orchestra.run.smithi176.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-29T11:15:52.206 INFO:teuthology.orchestra.run.smithi176.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-29T11:15:52.206 INFO:teuthology.orchestra.run.smithi176.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-29T11:15:52.206 INFO:teuthology.orchestra.run.smithi176.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-29T11:15:52.207 INFO:teuthology.orchestra.run.smithi176.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-29T11:15:52.207 INFO:teuthology.orchestra.run.smithi176.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-29T11:15:52.207 INFO:teuthology.orchestra.run.smithi176.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-29T11:15:52.207 INFO:teuthology.orchestra.run.smithi176.stdout: python3-ceph-argparse 2024-03-29T11:15:52.207 INFO:teuthology.orchestra.run.smithi176.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-29T11:15:52.207 INFO:teuthology.orchestra.run.smithi176.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-29T11:15:52.207 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-29T11:15:52.207 INFO:teuthology.orchestra.run.smithi176.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-29T11:15:52.207 INFO:teuthology.orchestra.run.smithi176.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-29T11:15:52.207 INFO:teuthology.orchestra.run.smithi176.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-29T11:15:52.207 INFO:teuthology.orchestra.run.smithi176.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-29T11:15:52.207 INFO:teuthology.orchestra.run.smithi176.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-03-29T11:15:52.207 INFO:teuthology.orchestra.run.smithi176.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-03-29T11:15:52.207 INFO:teuthology.orchestra.run.smithi176.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-29T11:15:52.207 INFO:teuthology.orchestra.run.smithi176.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-29T11:15:52.208 INFO:teuthology.orchestra.run.smithi176.stdout:Installing weak dependencies: 2024-03-29T11:15:52.208 INFO:teuthology.orchestra.run.smithi176.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-03-29T11:15:52.208 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T11:15:52.208 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-03-29T11:15:52.208 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-29T11:15:52.208 INFO:teuthology.orchestra.run.smithi176.stdout:Install 30 Packages 2024-03-29T11:15:52.208 INFO:teuthology.orchestra.run.smithi176.stdout:Upgrade 2 Packages 2024-03-29T11:15:52.208 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T11:15:52.208 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 58 M 2024-03-29T11:15:52.209 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-03-29T11:15:53.951 INFO:teuthology.orchestra.run.smithi176.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 3.8 MB/s | 5.3 MB 00:01 2024-03-29T11:15:53.993 INFO:teuthology.orchestra.run.smithi176.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 606 kB/s | 25 kB 00:00 2024-03-29T11:15:54.110 INFO:teuthology.orchestra.run.smithi176.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 6.2 MB/s | 742 kB 00:00 2024-03-29T11:15:54.211 INFO:teuthology.orchestra.run.smithi176.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 4.9 MB/s | 498 kB 00:00 2024-03-29T11:15:54.603 INFO:teuthology.orchestra.run.smithi176.stdout:(5/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 4.8 MB/s | 9.7 MB 00:02 2024-03-29T11:15:54.645 INFO:teuthology.orchestra.run.smithi176.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 1.1 MB/s | 45 kB 00:00 2024-03-29T11:15:54.695 INFO:teuthology.orchestra.run.smithi176.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 2.6 MB/s | 131 kB 00:00 2024-03-29T11:15:54.745 INFO:teuthology.orchestra.run.smithi176.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 3.2 MB/s | 162 kB 00:00 2024-03-29T11:15:54.837 INFO:teuthology.orchestra.run.smithi176.stdout:(9/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_6 8.1 MB/s | 5.1 MB 00:00 2024-03-29T11:15:54.862 INFO:teuthology.orchestra.run.smithi176.stdout:(10/32): python3-rados-19.0.0-1439.g5dcb190d.el 2.7 MB/s | 320 kB 00:00 2024-03-29T11:15:54.921 INFO:teuthology.orchestra.run.smithi176.stdout:(11/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 3.5 MB/s | 298 kB 00:00 2024-03-29T11:15:54.954 INFO:teuthology.orchestra.run.smithi176.stdout:(12/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 1.1 MB/s | 99 kB 00:00 2024-03-29T11:15:55.072 INFO:teuthology.orchestra.run.smithi176.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 703 kB/s | 106 kB 00:00 2024-03-29T11:15:55.097 INFO:teuthology.orchestra.run.smithi176.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.1 MB/s | 159 kB 00:00 2024-03-29T11:15:55.122 INFO:teuthology.orchestra.run.smithi176.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 883 kB/s | 44 kB 00:00 2024-03-29T11:15:55.181 INFO:teuthology.orchestra.run.smithi176.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 7.7 MB/s | 661 kB 00:00 2024-03-29T11:15:55.206 INFO:teuthology.orchestra.run.smithi176.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.3 MB/s | 282 kB 00:00 2024-03-29T11:15:55.231 INFO:teuthology.orchestra.run.smithi176.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 827 kB/s | 41 kB 00:00 2024-03-29T11:15:55.265 INFO:teuthology.orchestra.run.smithi176.stdout:(19/32): qatlib-24.02.0-1.el9.x86_64.rpm 3.7 MB/s | 220 kB 00:00 2024-03-29T11:15:55.290 INFO:teuthology.orchestra.run.smithi176.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 597 kB/s | 35 kB 00:00 2024-03-29T11:15:55.315 INFO:teuthology.orchestra.run.smithi176.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 923 kB/s | 46 kB 00:00 2024-03-29T11:15:55.640 INFO:teuthology.orchestra.run.smithi176.stdout:(22/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 882 kB/s | 309 kB 00:00 2024-03-29T11:15:55.682 INFO:teuthology.orchestra.run.smithi176.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 621 kB/s | 26 kB 00:00 2024-03-29T11:15:55.733 INFO:teuthology.orchestra.run.smithi176.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 971 kB/s | 49 kB 00:00 2024-03-29T11:15:55.774 INFO:teuthology.orchestra.run.smithi176.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.6 MB/s | 67 kB 00:00 2024-03-29T11:15:55.958 INFO:teuthology.orchestra.run.smithi176.stdout:(26/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 4.5 MB/s | 838 kB 00:00 2024-03-29T11:15:56.017 INFO:teuthology.orchestra.run.smithi176.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 3.2 MB/s | 192 kB 00:00 2024-03-29T11:15:56.350 INFO:teuthology.orchestra.run.smithi176.stdout:(28/32): thrift-0.15.0-2.el9.x86_64.rpm 4.7 MB/s | 1.6 MB 00:00 2024-03-29T11:15:56.517 INFO:teuthology.orchestra.run.smithi176.stdout:(29/32): libarrow-9.0.0-10.el9.x86_64.rpm 3.7 MB/s | 4.4 MB 00:01 2024-03-29T11:15:57.459 INFO:teuthology.orchestra.run.smithi176.stdout:(30/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 3.0 MB/s | 3.4 MB 00:01 2024-03-29T11:15:57.701 INFO:teuthology.orchestra.run.smithi176.stdout:(31/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 2.6 MB/s | 3.1 MB 00:01 2024-03-29T11:15:58.195 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - AppStream 878 kB/s | 19 MB 00:22 2024-03-29T11:16:01.801 INFO:teuthology.orchestra.run.smithi176.stdout:(32/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 2.2 MB/s | 20 MB 00:09 2024-03-29T11:16:01.805 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:16:01.806 INFO:teuthology.orchestra.run.smithi176.stdout:Total 6.0 MB/s | 58 MB 00:09 2024-03-29T11:16:01.806 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-03-29T11:16:01.844 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-03-29T11:16:01.844 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-03-29T11:16:02.309 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-03-29T11:16:02.310 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-03-29T11:16:02.729 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - CRB 6.1 MB/s | 6.1 MB 00:01 2024-03-29T11:16:03.785 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-03-29T11:16:03.866 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-29T11:16:03.918 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-29T11:16:04.100 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-29T11:16:04.129 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:16:04.146 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:16:04.287 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:16:04.341 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:16:04.388 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 9 - Extras packages 32 kB/s | 16 kB 00:00 2024-03-29T11:16:04.734 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-29T11:16:04.765 INFO:teuthology.orchestra.run.smithi176.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:16:04.866 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:16:04.956 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-29T11:16:04.983 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:16:05.046 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:16:05.075 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-29T11:16:05.119 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-29T11:16:05.188 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:16:05.211 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:16:05.274 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:16:05.337 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-29T11:16:05.445 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-29T11:16:05.494 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-29T11:16:05.534 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-29T11:16:05.567 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-29T11:16:05.596 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-29T11:16:05.637 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-29T11:16:05.948 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-29T11:16:05.975 INFO:teuthology.orchestra.run.smithi176.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:16:06.048 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:16:06.079 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-29T11:16:06.118 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-29T11:16:06.571 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-29T11:16:06.594 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:16:06.648 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:16:06.708 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-29T11:16:06.738 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-29T11:16:08.948 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:16:08.985 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:16:09.546 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:16:09.588 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:16:09.663 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:16:09.663 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-29T11:16:09.663 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-29T11:16:09.664 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T11:16:09.697 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:16:19.153 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:16:19.153 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /sys 2024-03-29T11:16:19.153 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /proc 2024-03-29T11:16:19.153 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /mnt 2024-03-29T11:16:19.153 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /var/tmp 2024-03-29T11:16:19.153 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /home 2024-03-29T11:16:19.154 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /root 2024-03-29T11:16:19.154 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /tmp 2024-03-29T11:16:19.154 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T11:16:20.051 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:16:20.075 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:16:20.075 INFO:teuthology.orchestra.run.smithi176.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:16:20.075 INFO:teuthology.orchestra.run.smithi176.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-29T11:16:20.075 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:16:20.075 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:16:20.075 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T11:16:20.077 INFO:teuthology.orchestra.run.smithi176.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:16:20.144 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:16:20.145 INFO:teuthology.orchestra.run.smithi176.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:16:21.264 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:16:21.264 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-29T11:16:21.264 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-29T11:16:21.265 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-29T11:16:21.265 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-29T11:16:21.265 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-29T11:16:21.265 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-29T11:16:21.265 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:16:21.265 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-29T11:16:21.265 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-29T11:16:21.265 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-29T11:16:21.265 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-29T11:16:21.265 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:16:21.265 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-29T11:16:21.265 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-29T11:16:21.265 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-29T11:16:21.265 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-29T11:16:21.266 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-29T11:16:21.266 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-29T11:16:21.266 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-29T11:16:21.266 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-29T11:16:21.266 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-29T11:16:21.266 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-29T11:16:21.266 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-29T11:16:21.267 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-29T11:16:21.267 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-29T11:16:21.267 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-29T11:16:21.267 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-29T11:16:21.267 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-29T11:16:21.267 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-29T11:16:21.267 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-29T11:16:21.268 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:16:21.268 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-29T11:16:21.268 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-29T11:16:22.464 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:16:22.465 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T11:16:22.465 INFO:teuthology.orchestra.run.smithi176.stdout:Upgraded: 2024-03-29T11:16:22.465 INFO:teuthology.orchestra.run.smithi176.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:22.465 INFO:teuthology.orchestra.run.smithi176.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:22.465 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-03-29T11:16:22.465 INFO:teuthology.orchestra.run.smithi176.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-29T11:16:22.465 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:22.465 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:22.465 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:22.465 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:22.465 INFO:teuthology.orchestra.run.smithi176.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-29T11:16:22.465 INFO:teuthology.orchestra.run.smithi176.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-29T11:16:22.465 INFO:teuthology.orchestra.run.smithi176.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-29T11:16:22.466 INFO:teuthology.orchestra.run.smithi176.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:22.466 INFO:teuthology.orchestra.run.smithi176.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-29T11:16:22.466 INFO:teuthology.orchestra.run.smithi176.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-29T11:16:22.466 INFO:teuthology.orchestra.run.smithi176.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-29T11:16:22.466 INFO:teuthology.orchestra.run.smithi176.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:22.466 INFO:teuthology.orchestra.run.smithi176.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-29T11:16:22.466 INFO:teuthology.orchestra.run.smithi176.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:22.466 INFO:teuthology.orchestra.run.smithi176.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-29T11:16:22.466 INFO:teuthology.orchestra.run.smithi176.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-29T11:16:22.466 INFO:teuthology.orchestra.run.smithi176.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-29T11:16:22.466 INFO:teuthology.orchestra.run.smithi176.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:22.466 INFO:teuthology.orchestra.run.smithi176.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:22.466 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:22.466 INFO:teuthology.orchestra.run.smithi176.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-29T11:16:22.467 INFO:teuthology.orchestra.run.smithi176.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:22.467 INFO:teuthology.orchestra.run.smithi176.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:22.467 INFO:teuthology.orchestra.run.smithi176.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:22.467 INFO:teuthology.orchestra.run.smithi176.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-29T11:16:22.467 INFO:teuthology.orchestra.run.smithi176.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-29T11:16:22.467 INFO:teuthology.orchestra.run.smithi176.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-29T11:16:22.467 INFO:teuthology.orchestra.run.smithi176.stdout: re2-1:20211101-3.el9.x86_64 2024-03-29T11:16:22.467 INFO:teuthology.orchestra.run.smithi176.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-29T11:16:22.467 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T11:16:22.467 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-29T11:16:22.829 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph-test 2024-03-29T11:16:23.400 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:00:34 ago on Fri 29 Mar 2024 11:15:49 AM UTC. 2024-03-29T11:16:23.565 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-29T11:16:23.566 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-29T11:16:23.566 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repository Size 2024-03-29T11:16:23.566 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-29T11:16:23.566 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-03-29T11:16:23.566 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-29T11:16:23.567 INFO:teuthology.orchestra.run.smithi176.stdout:Installing dependencies: 2024-03-29T11:16:23.567 INFO:teuthology.orchestra.run.smithi176.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-29T11:16:23.567 INFO:teuthology.orchestra.run.smithi176.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-29T11:16:23.567 INFO:teuthology.orchestra.run.smithi176.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-29T11:16:23.567 INFO:teuthology.orchestra.run.smithi176.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-29T11:16:23.567 INFO:teuthology.orchestra.run.smithi176.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-29T11:16:23.567 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T11:16:23.567 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-03-29T11:16:23.567 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-29T11:16:23.567 INFO:teuthology.orchestra.run.smithi176.stdout:Install 6 Packages 2024-03-29T11:16:23.567 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T11:16:23.568 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 48 M 2024-03-29T11:16:23.568 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 204 M 2024-03-29T11:16:23.568 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-03-29T11:16:23.798 INFO:teuthology.orchestra.run.smithi176.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.3 MB/s | 217 kB 00:00 2024-03-29T11:16:23.840 INFO:teuthology.orchestra.run.smithi176.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 7.0 MB/s | 300 kB 00:00 2024-03-29T11:16:23.899 INFO:teuthology.orchestra.run.smithi176.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.0 MB/s | 63 kB 00:00 2024-03-29T11:16:24.299 INFO:teuthology.orchestra.run.smithi176.stdout:(4/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 244 kB/s | 163 kB 00:00 2024-03-29T11:16:24.407 INFO:teuthology.orchestra.run.smithi176.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 368 kB/s | 187 kB 00:00 2024-03-29T11:17:04.762 INFO:teuthology.orchestra.run.smithi176.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 1.1 MB/s | 47 MB 00:41 2024-03-29T11:17:04.763 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:17:04.764 INFO:teuthology.orchestra.run.smithi176.stdout:Total 1.2 MB/s | 48 MB 00:41 2024-03-29T11:17:04.775 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-03-29T11:17:04.794 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-03-29T11:17:04.794 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-03-29T11:17:05.175 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-03-29T11:17:05.175 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-03-29T11:17:05.750 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-03-29T11:17:05.828 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-29T11:17:05.874 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-29T11:17:05.914 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:17:05.958 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-29T11:17:05.989 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:17:10.464 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:17:10.622 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:17:12.937 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:17:12.938 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-29T11:17:12.938 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-29T11:17:12.938 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:17:12.938 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-29T11:17:12.938 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-29T11:17:13.342 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-29T11:17:13.342 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T11:17:13.342 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-03-29T11:17:13.342 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:13.342 INFO:teuthology.orchestra.run.smithi176.stdout: jq-1.6-16.el9.x86_64 2024-03-29T11:17:13.343 INFO:teuthology.orchestra.run.smithi176.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:13.343 INFO:teuthology.orchestra.run.smithi176.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-29T11:17:13.343 INFO:teuthology.orchestra.run.smithi176.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-29T11:17:13.343 INFO:teuthology.orchestra.run.smithi176.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-29T11:17:13.343 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T11:17:13.343 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-29T11:17:13.659 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph 2024-03-29T11:17:14.228 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:01:25 ago on Fri 29 Mar 2024 11:15:49 AM UTC. 2024-03-29T11:17:14.386 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-29T11:17:14.388 INFO:teuthology.orchestra.run.smithi176.stdout:========================================================================================= 2024-03-29T11:17:14.388 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repository Size 2024-03-29T11:17:14.388 INFO:teuthology.orchestra.run.smithi176.stdout:========================================================================================= 2024-03-29T11:17:14.389 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-03-29T11:17:14.389 INFO:teuthology.orchestra.run.smithi176.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-29T11:17:14.389 INFO:teuthology.orchestra.run.smithi176.stdout:Installing dependencies: 2024-03-29T11:17:14.389 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-29T11:17:14.389 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-29T11:17:14.389 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-29T11:17:14.389 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-29T11:17:14.389 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-29T11:17:14.389 INFO:teuthology.orchestra.run.smithi176.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-29T11:17:14.389 INFO:teuthology.orchestra.run.smithi176.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-29T11:17:14.389 INFO:teuthology.orchestra.run.smithi176.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-29T11:17:14.389 INFO:teuthology.orchestra.run.smithi176.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-29T11:17:14.389 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-29T11:17:14.390 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-29T11:17:14.390 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-29T11:17:14.390 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-29T11:17:14.390 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-29T11:17:14.390 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-29T11:17:14.390 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-29T11:17:14.390 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-29T11:17:14.390 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-29T11:17:14.390 INFO:teuthology.orchestra.run.smithi176.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-29T11:17:14.390 INFO:teuthology.orchestra.run.smithi176.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-29T11:17:14.390 INFO:teuthology.orchestra.run.smithi176.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-29T11:17:14.390 INFO:teuthology.orchestra.run.smithi176.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-29T11:17:14.390 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-29T11:17:14.391 INFO:teuthology.orchestra.run.smithi176.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-29T11:17:14.391 INFO:teuthology.orchestra.run.smithi176.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-29T11:17:14.391 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-29T11:17:14.391 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-29T11:17:14.391 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-29T11:17:14.391 INFO:teuthology.orchestra.run.smithi176.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-29T11:17:14.391 INFO:teuthology.orchestra.run.smithi176.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-29T11:17:14.391 INFO:teuthology.orchestra.run.smithi176.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-29T11:17:14.391 INFO:teuthology.orchestra.run.smithi176.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-29T11:17:14.391 INFO:teuthology.orchestra.run.smithi176.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-29T11:17:14.391 INFO:teuthology.orchestra.run.smithi176.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-29T11:17:14.391 INFO:teuthology.orchestra.run.smithi176.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-29T11:17:14.391 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T11:17:14.391 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-03-29T11:17:14.392 INFO:teuthology.orchestra.run.smithi176.stdout:========================================================================================= 2024-03-29T11:17:14.392 INFO:teuthology.orchestra.run.smithi176.stdout:Install 36 Packages 2024-03-29T11:17:14.392 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T11:17:14.392 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 30 M 2024-03-29T11:17:14.392 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 107 M 2024-03-29T11:17:14.392 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-03-29T11:17:15.198 INFO:teuthology.orchestra.run.smithi176.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 26 kB/s | 6.6 kB 00:00 2024-03-29T11:17:15.881 INFO:teuthology.orchestra.run.smithi176.stdout:(2/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 2.3 MB/s | 2.1 MB 00:00 2024-03-29T11:17:16.207 INFO:teuthology.orchestra.run.smithi176.stdout:(3/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 1.2 MB/s | 1.5 MB 00:01 2024-03-29T11:17:16.382 INFO:teuthology.orchestra.run.smithi176.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 1.4 MB/s | 247 kB 00:00 2024-03-29T11:17:16.492 INFO:teuthology.orchestra.run.smithi176.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.7 MB/s | 187 kB 00:00 2024-03-29T11:17:16.535 INFO:teuthology.orchestra.run.smithi176.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-29T11:17:16.559 INFO:teuthology.orchestra.run.smithi176.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 1.2 MB/s | 32 kB 00:00 2024-03-29T11:17:16.585 INFO:teuthology.orchestra.run.smithi176.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.8 MB/s | 47 kB 00:00 2024-03-29T11:17:16.610 INFO:teuthology.orchestra.run.smithi176.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.5 MB/s | 39 kB 00:00 2024-03-29T11:17:16.652 INFO:teuthology.orchestra.run.smithi176.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.6 MB/s | 241 kB 00:00 2024-03-29T11:17:16.752 INFO:teuthology.orchestra.run.smithi176.stdout:(11/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 3.0 MB/s | 4.7 MB 00:01 2024-03-29T11:17:16.786 INFO:teuthology.orchestra.run.smithi176.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 8.6 MB/s | 1.1 MB 00:00 2024-03-29T11:17:16.811 INFO:teuthology.orchestra.run.smithi176.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 1.7 MB/s | 103 kB 00:00 2024-03-29T11:17:16.836 INFO:teuthology.orchestra.run.smithi176.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-03-29T11:17:16.862 INFO:teuthology.orchestra.run.smithi176.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.6 MB/s | 187 kB 00:00 2024-03-29T11:17:17.370 INFO:teuthology.orchestra.run.smithi176.stdout:(16/36): python3-requests-2.25.1-8.el9.noarch.r 234 kB/s | 125 kB 00:00 2024-03-29T11:17:17.454 INFO:teuthology.orchestra.run.smithi176.stdout:(17/36): lua-devel-5.4.4-4.el9.x86_64.rpm 38 kB/s | 22 kB 00:00 2024-03-29T11:17:17.579 INFO:teuthology.orchestra.run.smithi176.stdout:(18/36): luarocks-3.9.2-1.el9.noarch.rpm 725 kB/s | 151 kB 00:00 2024-03-29T11:17:17.596 INFO:teuthology.orchestra.run.smithi176.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 307 kB/s | 43 kB 00:00 2024-03-29T11:17:17.663 INFO:teuthology.orchestra.run.smithi176.stdout:(20/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.0 MB/s | 172 kB 00:00 2024-03-29T11:17:17.713 INFO:teuthology.orchestra.run.smithi176.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 214 kB/s | 11 kB 00:00 2024-03-29T11:17:17.872 INFO:teuthology.orchestra.run.smithi176.stdout:(22/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 8.4 MB/s | 17 MB 00:01 2024-03-29T11:17:17.897 INFO:teuthology.orchestra.run.smithi176.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 96 kB/s | 18 kB 00:00 2024-03-29T11:17:17.922 INFO:teuthology.orchestra.run.smithi176.stdout:(24/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.1 MB/s | 358 kB 00:00 2024-03-29T11:17:17.947 INFO:teuthology.orchestra.run.smithi176.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 386 kB/s | 19 kB 00:00 2024-03-29T11:17:17.973 INFO:teuthology.orchestra.run.smithi176.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 390 kB/s | 20 kB 00:00 2024-03-29T11:17:18.007 INFO:teuthology.orchestra.run.smithi176.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 788 kB/s | 46 kB 00:00 2024-03-29T11:17:18.032 INFO:teuthology.orchestra.run.smithi176.stdout:(28/36): python3-jaraco-collections-3.0.0-8.el9 145 kB/s | 23 kB 00:00 2024-03-29T11:17:18.057 INFO:teuthology.orchestra.run.smithi176.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 941 kB/s | 79 kB 00:00 2024-03-29T11:17:18.082 INFO:teuthology.orchestra.run.smithi176.stdout:(30/36): python3-pecan-1.4.2-3.el9.noarch.rpm 3.5 MB/s | 272 kB 00:00 2024-03-29T11:17:18.108 INFO:teuthology.orchestra.run.smithi176.stdout:(31/36): python3-portend-3.1.0-2.el9.noarch.rpm 217 kB/s | 16 kB 00:00 2024-03-29T11:17:18.125 INFO:teuthology.orchestra.run.smithi176.stdout:(32/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.3 MB/s | 90 kB 00:00 2024-03-29T11:17:18.150 INFO:teuthology.orchestra.run.smithi176.stdout:(33/36): python3-tempora-5.0.0-2.el9.noarch.rpm 531 kB/s | 36 kB 00:00 2024-03-29T11:17:18.200 INFO:teuthology.orchestra.run.smithi176.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 401 kB/s | 20 kB 00:00 2024-03-29T11:17:18.225 INFO:teuthology.orchestra.run.smithi176.stdout:(35/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.2 MB/s | 427 kB 00:00 2024-03-29T11:17:18.259 INFO:teuthology.orchestra.run.smithi176.stdout:(36/36): python3-webob-1.8.7-6.el9.noarch.rpm 1.5 MB/s | 230 kB 00:00 2024-03-29T11:17:18.262 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:17:18.262 INFO:teuthology.orchestra.run.smithi176.stdout:Total 7.7 MB/s | 30 MB 00:03 2024-03-29T11:17:18.284 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-03-29T11:17:18.323 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-03-29T11:17:18.323 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-03-29T11:17:18.741 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-03-29T11:17:18.742 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-03-29T11:17:20.102 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-03-29T11:17:20.170 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-29T11:17:20.209 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-29T11:17:20.239 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-29T11:17:20.310 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-29T11:17:20.354 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-29T11:17:20.387 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-29T11:17:20.478 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-29T11:17:20.557 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-29T11:17:20.598 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-29T11:17:20.669 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-29T11:17:20.724 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-29T11:17:20.762 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-29T11:17:20.811 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-29T11:17:20.865 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-29T11:17:21.051 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-29T11:17:21.099 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-29T11:17:21.138 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-29T11:17:21.172 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-29T11:17:21.202 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-29T11:17:21.278 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-29T11:17:21.312 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:17:21.346 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-29T11:17:21.386 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-29T11:17:21.421 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-29T11:17:21.464 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-29T11:17:21.553 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-29T11:17:21.586 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-29T11:17:21.659 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-29T11:17:21.795 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-29T11:17:21.912 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:17:22.070 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-29T11:17:22.099 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:17:22.125 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:17:22.125 INFO:teuthology.orchestra.run.smithi176.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:17:22.125 INFO:teuthology.orchestra.run.smithi176.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-29T11:17:22.125 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:17:22.125 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:17:22.125 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T11:17:23.550 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:17:23.577 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:17:23.577 INFO:teuthology.orchestra.run.smithi176.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:17:23.577 INFO:teuthology.orchestra.run.smithi176.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-29T11:17:23.577 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:17:23.577 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:17:23.577 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T11:17:24.018 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:17:24.041 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:17:24.041 INFO:teuthology.orchestra.run.smithi176.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:17:24.041 INFO:teuthology.orchestra.run.smithi176.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-29T11:17:24.041 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:17:24.041 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:17:24.041 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T11:17:24.252 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:17:24.275 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:17:24.275 INFO:teuthology.orchestra.run.smithi176.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:17:24.275 INFO:teuthology.orchestra.run.smithi176.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-29T11:17:24.275 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:17:24.275 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:17:24.275 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T11:17:24.341 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:17:26.131 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:17:26.131 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-29T11:17:26.131 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-29T11:17:26.131 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-29T11:17:26.131 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-29T11:17:26.131 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-29T11:17:26.132 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-29T11:17:26.132 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-29T11:17:26.132 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-29T11:17:26.132 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-29T11:17:26.132 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-29T11:17:26.132 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-29T11:17:26.132 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-29T11:17:26.132 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-29T11:17:26.132 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-29T11:17:26.132 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-29T11:17:26.132 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-29T11:17:26.132 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-29T11:17:26.133 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-29T11:17:26.133 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-29T11:17:26.133 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-29T11:17:26.133 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:17:26.133 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-29T11:17:26.133 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-29T11:17:26.133 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-29T11:17:26.133 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-29T11:17:26.133 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-29T11:17:26.133 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-29T11:17:26.133 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-29T11:17:26.133 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-29T11:17:26.133 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:17:26.134 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-29T11:17:26.134 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-29T11:17:26.134 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-29T11:17:26.134 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-29T11:17:26.134 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-29T11:17:27.103 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-29T11:17:27.104 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T11:17:27.104 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-03-29T11:17:27.104 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:27.104 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:27.104 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:27.104 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:17:27.104 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:27.104 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:27.104 INFO:teuthology.orchestra.run.smithi176.stdout: lua-5.4.4-4.el9.x86_64 2024-03-29T11:17:27.104 INFO:teuthology.orchestra.run.smithi176.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-29T11:17:27.104 INFO:teuthology.orchestra.run.smithi176.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-29T11:17:27.104 INFO:teuthology.orchestra.run.smithi176.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-29T11:17:27.104 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-29T11:17:27.104 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-29T11:17:27.105 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-29T11:17:27.105 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-29T11:17:27.105 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-29T11:17:27.105 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-29T11:17:27.105 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-29T11:17:27.105 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-29T11:17:27.105 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-29T11:17:27.105 INFO:teuthology.orchestra.run.smithi176.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-29T11:17:27.105 INFO:teuthology.orchestra.run.smithi176.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-29T11:17:27.105 INFO:teuthology.orchestra.run.smithi176.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-29T11:17:27.105 INFO:teuthology.orchestra.run.smithi176.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-29T11:17:27.105 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-29T11:17:27.105 INFO:teuthology.orchestra.run.smithi176.stdout: python3-ply-3.11-14.el9.noarch 2024-03-29T11:17:27.105 INFO:teuthology.orchestra.run.smithi176.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-29T11:17:27.105 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-29T11:17:27.106 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-29T11:17:27.106 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-29T11:17:27.106 INFO:teuthology.orchestra.run.smithi176.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-29T11:17:27.106 INFO:teuthology.orchestra.run.smithi176.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-29T11:17:27.106 INFO:teuthology.orchestra.run.smithi176.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-29T11:17:27.106 INFO:teuthology.orchestra.run.smithi176.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-29T11:17:27.106 INFO:teuthology.orchestra.run.smithi176.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-29T11:17:27.106 INFO:teuthology.orchestra.run.smithi176.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-29T11:17:27.106 INFO:teuthology.orchestra.run.smithi176.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-29T11:17:27.106 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T11:17:27.106 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-29T11:17:27.466 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph-base 2024-03-29T11:17:28.046 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:01:39 ago on Fri 29 Mar 2024 11:15:49 AM UTC. 2024-03-29T11:17:28.143 INFO:teuthology.orchestra.run.smithi176.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:17:28.195 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-29T11:17:28.196 INFO:teuthology.orchestra.run.smithi176.stdout:Nothing to do. 2024-03-29T11:17:28.197 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-29T11:17:28.262 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install cephadm 2024-03-29T11:17:28.825 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:01:39 ago on Fri 29 Mar 2024 11:15:49 AM UTC. 2024-03-29T11:17:28.974 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-29T11:17:28.975 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-29T11:17:28.975 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repository Size 2024-03-29T11:17:28.975 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-29T11:17:28.975 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-03-29T11:17:28.975 INFO:teuthology.orchestra.run.smithi176.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-29T11:17:28.976 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T11:17:28.976 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-03-29T11:17:28.976 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-29T11:17:28.976 INFO:teuthology.orchestra.run.smithi176.stdout:Install 1 Package 2024-03-29T11:17:28.976 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T11:17:28.976 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 747 k 2024-03-29T11:17:28.976 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 754 k 2024-03-29T11:17:28.976 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-03-29T11:17:30.009 INFO:teuthology.orchestra.run.smithi176.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 724 kB/s | 747 kB 00:01 2024-03-29T11:17:30.009 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:17:30.010 INFO:teuthology.orchestra.run.smithi176.stdout:Total 723 kB/s | 747 kB 00:01 2024-03-29T11:17:30.010 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-03-29T11:17:30.015 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-03-29T11:17:30.015 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-03-29T11:17:30.021 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-03-29T11:17:30.021 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-03-29T11:17:30.101 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-03-29T11:17:30.492 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:17:30.589 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:17:31.005 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:17:31.294 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:17:31.294 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T11:17:31.294 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-03-29T11:17:31.295 INFO:teuthology.orchestra.run.smithi176.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:17:31.295 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T11:17:31.295 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-29T11:17:31.521 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph-immutable-object-cache 2024-03-29T11:17:32.094 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:01:43 ago on Fri 29 Mar 2024 11:15:49 AM UTC. 2024-03-29T11:17:32.249 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-29T11:17:32.250 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-29T11:17:32.250 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repo Size 2024-03-29T11:17:32.250 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-29T11:17:32.250 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-03-29T11:17:32.250 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-29T11:17:32.250 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T11:17:32.250 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-03-29T11:17:32.250 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-29T11:17:32.250 INFO:teuthology.orchestra.run.smithi176.stdout:Install 1 Package 2024-03-29T11:17:32.250 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T11:17:32.250 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 145 k 2024-03-29T11:17:32.250 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 436 k 2024-03-29T11:17:32.250 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-03-29T11:17:33.524 INFO:teuthology.orchestra.run.smithi176.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 114 kB/s | 145 kB 00:01 2024-03-29T11:17:33.524 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:17:33.524 INFO:teuthology.orchestra.run.smithi176.stdout:Total 114 kB/s | 145 kB 00:01 2024-03-29T11:17:33.525 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-03-29T11:17:33.534 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-03-29T11:17:33.534 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-03-29T11:17:33.591 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-03-29T11:17:33.591 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-03-29T11:17:33.731 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-03-29T11:17:33.799 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:17:33.825 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:17:33.825 INFO:teuthology.orchestra.run.smithi176.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:17:33.825 INFO:teuthology.orchestra.run.smithi176.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-29T11:17:33.825 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T11:17:34.780 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:17:34.780 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T11:17:34.780 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-03-29T11:17:34.780 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:34.780 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T11:17:34.780 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-29T11:17:35.009 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph-mgr 2024-03-29T11:17:35.588 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:01:46 ago on Fri 29 Mar 2024 11:15:49 AM UTC. 2024-03-29T11:17:35.689 INFO:teuthology.orchestra.run.smithi176.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:17:35.741 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-29T11:17:35.742 INFO:teuthology.orchestra.run.smithi176.stdout:Nothing to do. 2024-03-29T11:17:35.742 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-29T11:17:35.813 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph-mgr-dashboard 2024-03-29T11:17:36.376 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:01:47 ago on Fri 29 Mar 2024 11:15:49 AM UTC. 2024-03-29T11:17:36.526 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-29T11:17:36.527 INFO:teuthology.orchestra.run.smithi176.stdout:====================================================================================== 2024-03-29T11:17:36.527 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repository Size 2024-03-29T11:17:36.527 INFO:teuthology.orchestra.run.smithi176.stdout:====================================================================================== 2024-03-29T11:17:36.527 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-03-29T11:17:36.527 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-29T11:17:36.527 INFO:teuthology.orchestra.run.smithi176.stdout:Installing dependencies: 2024-03-29T11:17:36.527 INFO:teuthology.orchestra.run.smithi176.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-29T11:17:36.527 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-29T11:17:36.528 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-29T11:17:36.528 INFO:teuthology.orchestra.run.smithi176.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-29T11:17:36.528 INFO:teuthology.orchestra.run.smithi176.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-29T11:17:36.528 INFO:teuthology.orchestra.run.smithi176.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-29T11:17:36.528 INFO:teuthology.orchestra.run.smithi176.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-29T11:17:36.528 INFO:teuthology.orchestra.run.smithi176.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-29T11:17:36.528 INFO:teuthology.orchestra.run.smithi176.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-29T11:17:36.528 INFO:teuthology.orchestra.run.smithi176.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-29T11:17:36.528 INFO:teuthology.orchestra.run.smithi176.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-29T11:17:36.528 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T11:17:36.528 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-03-29T11:17:36.528 INFO:teuthology.orchestra.run.smithi176.stdout:====================================================================================== 2024-03-29T11:17:36.528 INFO:teuthology.orchestra.run.smithi176.stdout:Install 12 Packages 2024-03-29T11:17:36.528 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T11:17:36.529 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 6.9 M 2024-03-29T11:17:36.529 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 26 M 2024-03-29T11:17:36.529 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-03-29T11:17:37.277 INFO:teuthology.orchestra.run.smithi176.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1439.g5d 60 kB/s | 31 kB 00:00 2024-03-29T11:17:37.294 INFO:teuthology.orchestra.run.smithi176.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 29 kB/s | 15 kB 00:00 2024-03-29T11:17:37.436 INFO:teuthology.orchestra.run.smithi176.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.6 MB/s | 238 kB 00:00 2024-03-29T11:17:37.478 INFO:teuthology.orchestra.run.smithi176.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 5.0 MB/s | 1.0 MB 00:00 2024-03-29T11:17:38.070 INFO:teuthology.orchestra.run.smithi176.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 926 kB/s | 548 kB 00:00 2024-03-29T11:17:38.261 INFO:teuthology.orchestra.run.smithi176.stdout:(6/12): protobuf-compiler-3.14.0-13.el9.x86_64. 1.0 MB/s | 863 kB 00:00 2024-03-29T11:17:38.329 INFO:teuthology.orchestra.run.smithi176.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 75 kB/s | 19 kB 00:00 2024-03-29T11:17:38.479 INFO:teuthology.orchestra.run.smithi176.stdout:(8/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb190 1.0 MB/s | 1.8 MB 00:01 2024-03-29T11:17:38.737 INFO:teuthology.orchestra.run.smithi176.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 119 kB/s | 31 kB 00:00 2024-03-29T11:17:38.762 INFO:teuthology.orchestra.run.smithi176.stdout:(10/12): python3-grpcio-tools-1.46.7-10.el9.x86 333 kB/s | 144 kB 00:00 2024-03-29T11:17:39.104 INFO:teuthology.orchestra.run.smithi176.stdout:(11/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 2.4 MB/s | 2.0 MB 00:00 2024-03-29T11:17:39.163 INFO:teuthology.orchestra.run.smithi176.stdout:(12/12): python3-routes-2.5.1-5.el9.noarch.rpm 443 kB/s | 188 kB 00:00 2024-03-29T11:17:39.168 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:17:39.168 INFO:teuthology.orchestra.run.smithi176.stdout:Total 2.6 MB/s | 6.9 MB 00:02 2024-03-29T11:17:39.188 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-03-29T11:17:39.207 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-03-29T11:17:39.207 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-03-29T11:17:39.385 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-03-29T11:17:39.385 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-03-29T11:17:39.904 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-03-29T11:17:39.988 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-29T11:17:40.024 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-29T11:17:40.069 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-29T11:17:40.101 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-29T11:17:40.208 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-29T11:17:40.333 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-29T11:17:40.427 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-29T11:17:40.480 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-29T11:17:40.509 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:17:40.540 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-29T11:17:40.782 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-29T11:17:40.807 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:17:41.649 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:17:41.649 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-29T11:17:41.649 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-29T11:17:41.649 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-29T11:17:41.649 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-29T11:17:41.650 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-29T11:17:41.650 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-29T11:17:41.650 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-29T11:17:41.650 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-29T11:17:41.650 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:17:41.650 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-29T11:17:41.650 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-29T11:17:41.994 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-29T11:17:41.994 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T11:17:41.995 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-03-29T11:17:41.995 INFO:teuthology.orchestra.run.smithi176.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-29T11:17:41.995 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:17:41.995 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:17:41.995 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:17:41.995 INFO:teuthology.orchestra.run.smithi176.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-29T11:17:41.995 INFO:teuthology.orchestra.run.smithi176.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-29T11:17:41.995 INFO:teuthology.orchestra.run.smithi176.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-29T11:17:41.995 INFO:teuthology.orchestra.run.smithi176.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-29T11:17:41.995 INFO:teuthology.orchestra.run.smithi176.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-29T11:17:41.995 INFO:teuthology.orchestra.run.smithi176.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-29T11:17:41.995 INFO:teuthology.orchestra.run.smithi176.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-29T11:17:41.995 INFO:teuthology.orchestra.run.smithi176.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-29T11:17:41.995 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T11:17:41.996 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-29T11:17:42.269 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-29T11:17:42.832 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:01:53 ago on Fri 29 Mar 2024 11:15:49 AM UTC. 2024-03-29T11:17:42.985 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-29T11:17:42.986 INFO:teuthology.orchestra.run.smithi176.stdout:============================================================================================ 2024-03-29T11:17:42.986 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repository Size 2024-03-29T11:17:42.986 INFO:teuthology.orchestra.run.smithi176.stdout:============================================================================================ 2024-03-29T11:17:42.986 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-03-29T11:17:42.986 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-29T11:17:42.986 INFO:teuthology.orchestra.run.smithi176.stdout:Installing dependencies: 2024-03-29T11:17:42.986 INFO:teuthology.orchestra.run.smithi176.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-29T11:17:42.986 INFO:teuthology.orchestra.run.smithi176.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-29T11:17:42.987 INFO:teuthology.orchestra.run.smithi176.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-29T11:17:42.987 INFO:teuthology.orchestra.run.smithi176.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-29T11:17:42.987 INFO:teuthology.orchestra.run.smithi176.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-29T11:17:42.987 INFO:teuthology.orchestra.run.smithi176.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-29T11:17:42.987 INFO:teuthology.orchestra.run.smithi176.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-29T11:17:42.987 INFO:teuthology.orchestra.run.smithi176.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-29T11:17:42.987 INFO:teuthology.orchestra.run.smithi176.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-29T11:17:42.987 INFO:teuthology.orchestra.run.smithi176.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-29T11:17:42.987 INFO:teuthology.orchestra.run.smithi176.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-29T11:17:42.987 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T11:17:42.987 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-03-29T11:17:42.987 INFO:teuthology.orchestra.run.smithi176.stdout:============================================================================================ 2024-03-29T11:17:42.987 INFO:teuthology.orchestra.run.smithi176.stdout:Install 12 Packages 2024-03-29T11:17:42.987 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T11:17:42.988 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 37 M 2024-03-29T11:17:42.988 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 209 M 2024-03-29T11:17:42.988 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-03-29T11:17:43.511 INFO:teuthology.orchestra.run.smithi176.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 270 kB/s | 32 kB 00:00 2024-03-29T11:17:43.545 INFO:teuthology.orchestra.run.smithi176.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 512 kB/s | 17 kB 00:00 2024-03-29T11:17:43.578 INFO:teuthology.orchestra.run.smithi176.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2024-03-29T11:17:43.762 INFO:teuthology.orchestra.run.smithi176.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 26 MB/s | 4.7 MB 00:00 2024-03-29T11:17:43.912 INFO:teuthology.orchestra.run.smithi176.stdout:(5/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 32 MB/s | 4.8 MB 00:00 2024-03-29T11:17:43.954 INFO:teuthology.orchestra.run.smithi176.stdout:(6/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 5.3 MB/s | 217 kB 00:00 2024-03-29T11:17:44.021 INFO:teuthology.orchestra.run.smithi176.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 4.8 MB/s | 3.0 MB 00:00 2024-03-29T11:17:44.313 INFO:teuthology.orchestra.run.smithi176.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 44 MB/s | 16 MB 00:00 2024-03-29T11:17:44.463 INFO:teuthology.orchestra.run.smithi176.stdout:(9/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.3 MB/s | 195 kB 00:00 2024-03-29T11:17:44.496 INFO:teuthology.orchestra.run.smithi176.stdout:(10/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.7 MB/s | 803 kB 00:00 2024-03-29T11:17:44.622 INFO:teuthology.orchestra.run.smithi176.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.5 MB/s | 245 kB 00:00 2024-03-29T11:17:46.555 INFO:teuthology.orchestra.run.smithi176.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 2.3 MB/s | 7.4 MB 00:03 2024-03-29T11:17:46.558 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:17:46.558 INFO:teuthology.orchestra.run.smithi176.stdout:Total 10 MB/s | 37 MB 00:03 2024-03-29T11:17:46.602 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-03-29T11:17:46.628 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-03-29T11:17:46.628 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-03-29T11:17:46.986 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-03-29T11:17:46.987 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-03-29T11:17:47.678 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-03-29T11:17:47.752 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-29T11:17:47.784 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-29T11:17:47.868 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:17:47.904 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-29T11:17:48.080 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:17:48.103 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:17:48.180 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-29T11:17:48.830 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-29T11:17:48.942 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-29T11:17:50.197 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-29T11:17:51.216 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-29T11:17:51.244 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:17:53.650 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:17:53.650 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-29T11:17:53.651 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-29T11:17:53.651 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:17:53.651 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-29T11:17:53.651 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:17:53.651 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:17:53.651 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-29T11:17:53.651 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-29T11:17:53.651 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-29T11:17:53.651 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-29T11:17:53.651 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-29T11:17:54.089 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-29T11:17:54.090 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T11:17:54.090 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-03-29T11:17:54.090 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:17:54.090 INFO:teuthology.orchestra.run.smithi176.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-29T11:17:54.090 INFO:teuthology.orchestra.run.smithi176.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-29T11:17:54.090 INFO:teuthology.orchestra.run.smithi176.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-29T11:17:54.090 INFO:teuthology.orchestra.run.smithi176.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-29T11:17:54.091 INFO:teuthology.orchestra.run.smithi176.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-29T11:17:54.091 INFO:teuthology.orchestra.run.smithi176.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-29T11:17:54.091 INFO:teuthology.orchestra.run.smithi176.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-29T11:17:54.091 INFO:teuthology.orchestra.run.smithi176.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-29T11:17:54.091 INFO:teuthology.orchestra.run.smithi176.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-29T11:17:54.091 INFO:teuthology.orchestra.run.smithi176.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-29T11:17:54.091 INFO:teuthology.orchestra.run.smithi176.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-29T11:17:54.091 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T11:17:54.091 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-29T11:17:54.476 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph-mgr-rook 2024-03-29T11:17:55.058 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:02:06 ago on Fri 29 Mar 2024 11:15:49 AM UTC. 2024-03-29T11:17:55.222 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-29T11:17:55.223 INFO:teuthology.orchestra.run.smithi176.stdout:======================================================================================== 2024-03-29T11:17:55.223 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repository Size 2024-03-29T11:17:55.223 INFO:teuthology.orchestra.run.smithi176.stdout:======================================================================================== 2024-03-29T11:17:55.223 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-03-29T11:17:55.223 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-29T11:17:55.223 INFO:teuthology.orchestra.run.smithi176.stdout:Installing dependencies: 2024-03-29T11:17:55.223 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-29T11:17:55.224 INFO:teuthology.orchestra.run.smithi176.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-29T11:17:55.224 INFO:teuthology.orchestra.run.smithi176.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-29T11:17:55.224 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-29T11:17:55.224 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-29T11:17:55.224 INFO:teuthology.orchestra.run.smithi176.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-29T11:17:55.224 INFO:teuthology.orchestra.run.smithi176.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-29T11:17:55.224 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-29T11:17:55.224 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-29T11:17:55.224 INFO:teuthology.orchestra.run.smithi176.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-29T11:17:55.224 INFO:teuthology.orchestra.run.smithi176.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-29T11:17:55.224 INFO:teuthology.orchestra.run.smithi176.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-29T11:17:55.224 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T11:17:55.224 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-03-29T11:17:55.224 INFO:teuthology.orchestra.run.smithi176.stdout:======================================================================================== 2024-03-29T11:17:55.225 INFO:teuthology.orchestra.run.smithi176.stdout:Install 13 Packages 2024-03-29T11:17:55.225 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T11:17:55.225 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 2.0 M 2024-03-29T11:17:55.225 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 26 M 2024-03-29T11:17:55.225 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-03-29T11:17:55.394 INFO:teuthology.orchestra.run.smithi176.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 152 kB/s | 18 kB 00:00 2024-03-29T11:17:55.420 INFO:teuthology.orchestra.run.smithi176.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 170 kB/s | 24 kB 00:00 2024-03-29T11:17:55.462 INFO:teuthology.orchestra.run.smithi176.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.6 MB/s | 174 kB 00:00 2024-03-29T11:17:55.487 INFO:teuthology.orchestra.run.smithi176.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 132 kB 00:00 2024-03-29T11:17:55.512 INFO:teuthology.orchestra.run.smithi176.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-29T11:17:55.537 INFO:teuthology.orchestra.run.smithi176.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 856 kB/s | 43 kB 00:00 2024-03-29T11:17:55.562 INFO:teuthology.orchestra.run.smithi176.stdout:(7/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 182 kB/s | 52 kB 00:00 2024-03-29T11:17:55.788 INFO:teuthology.orchestra.run.smithi176.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 117 kB/s | 32 kB 00:00 2024-03-29T11:17:55.813 INFO:teuthology.orchestra.run.smithi176.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 54 kB/s | 15 kB 00:00 2024-03-29T11:17:55.971 INFO:teuthology.orchestra.run.smithi176.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 373 kB/s | 59 kB 00:00 2024-03-29T11:17:56.005 INFO:teuthology.orchestra.run.smithi176.stdout:(11/13): python3-google-auth-2.28.2-1.el9.noarc 458 kB/s | 203 kB 00:00 2024-03-29T11:17:56.097 INFO:teuthology.orchestra.run.smithi176.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 716 kB/s | 90 kB 00:00 2024-03-29T11:17:56.139 INFO:teuthology.orchestra.run.smithi176.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 2.9 MB/s | 1.0 MB 00:00 2024-03-29T11:17:56.143 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:17:56.143 INFO:teuthology.orchestra.run.smithi176.stdout:Total 2.2 MB/s | 2.0 MB 00:00 2024-03-29T11:17:56.143 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-03-29T11:17:56.157 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-03-29T11:17:56.157 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-03-29T11:17:56.250 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-03-29T11:17:56.251 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-03-29T11:17:56.756 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-03-29T11:17:56.918 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-29T11:17:56.982 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-29T11:17:57.054 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-29T11:17:57.100 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-29T11:17:57.141 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-29T11:17:57.234 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-29T11:17:57.353 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-29T11:17:57.409 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-29T11:17:58.125 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-29T11:17:58.181 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-29T11:17:58.224 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-29T11:17:58.272 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-29T11:17:58.294 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:17:59.117 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:17:59.117 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-29T11:17:59.117 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-29T11:17:59.117 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-29T11:17:59.117 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-29T11:17:59.117 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-29T11:17:59.117 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-29T11:17:59.117 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-29T11:17:59.117 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-29T11:17:59.117 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-29T11:17:59.117 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-29T11:17:59.118 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-29T11:17:59.118 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-29T11:17:59.475 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-29T11:17:59.475 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T11:17:59.475 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-03-29T11:17:59.475 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:17:59.475 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-29T11:17:59.475 INFO:teuthology.orchestra.run.smithi176.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-29T11:17:59.475 INFO:teuthology.orchestra.run.smithi176.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-29T11:17:59.476 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-29T11:17:59.476 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-29T11:17:59.476 INFO:teuthology.orchestra.run.smithi176.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-29T11:17:59.476 INFO:teuthology.orchestra.run.smithi176.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-29T11:17:59.476 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-29T11:17:59.476 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-29T11:17:59.476 INFO:teuthology.orchestra.run.smithi176.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-29T11:17:59.476 INFO:teuthology.orchestra.run.smithi176.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-29T11:17:59.476 INFO:teuthology.orchestra.run.smithi176.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-29T11:17:59.476 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T11:17:59.477 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-29T11:17:59.720 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph-mgr-cephadm 2024-03-29T11:18:00.291 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:02:11 ago on Fri 29 Mar 2024 11:15:49 AM UTC. 2024-03-29T11:18:00.445 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-29T11:18:00.446 INFO:teuthology.orchestra.run.smithi176.stdout:======================================================================================== 2024-03-29T11:18:00.446 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repository Size 2024-03-29T11:18:00.446 INFO:teuthology.orchestra.run.smithi176.stdout:======================================================================================== 2024-03-29T11:18:00.446 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-03-29T11:18:00.446 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-29T11:18:00.446 INFO:teuthology.orchestra.run.smithi176.stdout:Installing dependencies: 2024-03-29T11:18:00.446 INFO:teuthology.orchestra.run.smithi176.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-29T11:18:00.446 INFO:teuthology.orchestra.run.smithi176.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-29T11:18:00.446 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-29T11:18:00.446 INFO:teuthology.orchestra.run.smithi176.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-29T11:18:00.446 INFO:teuthology.orchestra.run.smithi176.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-29T11:18:00.447 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T11:18:00.447 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-03-29T11:18:00.447 INFO:teuthology.orchestra.run.smithi176.stdout:======================================================================================== 2024-03-29T11:18:00.447 INFO:teuthology.orchestra.run.smithi176.stdout:Install 6 Packages 2024-03-29T11:18:00.447 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T11:18:00.447 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 6.8 M 2024-03-29T11:18:00.448 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 33 M 2024-03-29T11:18:00.448 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-03-29T11:18:00.800 INFO:teuthology.orchestra.run.smithi176.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.3 MB/s | 248 kB 00:00 2024-03-29T11:18:00.875 INFO:teuthology.orchestra.run.smithi176.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 22 MB/s | 5.8 MB 00:00 2024-03-29T11:18:00.900 INFO:teuthology.orchestra.run.smithi176.stdout:(3/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 477 kB/s | 138 kB 00:00 2024-03-29T11:18:01.226 INFO:teuthology.orchestra.run.smithi176.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 165 kB/s | 58 kB 00:00 2024-03-29T11:18:01.251 INFO:teuthology.orchestra.run.smithi176.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 146 kB/s | 51 kB 00:00 2024-03-29T11:18:01.409 INFO:teuthology.orchestra.run.smithi176.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 903 kB/s | 548 kB 00:00 2024-03-29T11:18:01.411 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:01.411 INFO:teuthology.orchestra.run.smithi176.stdout:Total 7.0 MB/s | 6.8 MB 00:00 2024-03-29T11:18:01.424 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-03-29T11:18:01.434 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-03-29T11:18:01.434 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-03-29T11:18:01.523 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-03-29T11:18:01.524 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-03-29T11:18:01.834 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-03-29T11:18:02.007 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-29T11:18:02.052 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-29T11:18:02.571 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-29T11:18:02.646 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-29T11:18:02.716 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-29T11:18:02.740 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:18:03.504 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:18:03.504 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-29T11:18:03.504 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-29T11:18:03.504 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-29T11:18:03.504 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-29T11:18:03.504 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-29T11:18:03.801 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-29T11:18:03.801 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T11:18:03.801 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-03-29T11:18:03.801 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:18:03.801 INFO:teuthology.orchestra.run.smithi176.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-29T11:18:03.801 INFO:teuthology.orchestra.run.smithi176.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-29T11:18:03.801 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-29T11:18:03.801 INFO:teuthology.orchestra.run.smithi176.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-29T11:18:03.801 INFO:teuthology.orchestra.run.smithi176.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-29T11:18:03.801 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T11:18:03.801 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-29T11:18:04.048 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph-fuse 2024-03-29T11:18:04.612 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:02:15 ago on Fri 29 Mar 2024 11:15:49 AM UTC. 2024-03-29T11:18:04.762 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-29T11:18:04.763 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-29T11:18:04.763 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repository Size 2024-03-29T11:18:04.763 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-29T11:18:04.763 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-03-29T11:18:04.763 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-29T11:18:04.763 INFO:teuthology.orchestra.run.smithi176.stdout:Installing dependencies: 2024-03-29T11:18:04.763 INFO:teuthology.orchestra.run.smithi176.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-29T11:18:04.763 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T11:18:04.763 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-03-29T11:18:04.764 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-29T11:18:04.764 INFO:teuthology.orchestra.run.smithi176.stdout:Install 2 Packages 2024-03-29T11:18:04.764 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T11:18:04.764 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 964 k 2024-03-29T11:18:04.764 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 2.8 M 2024-03-29T11:18:04.764 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-03-29T11:18:04.885 INFO:teuthology.orchestra.run.smithi176.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 650 kB/s | 79 kB 00:00 2024-03-29T11:18:05.469 INFO:teuthology.orchestra.run.smithi176.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 1.2 MB/s | 885 kB 00:00 2024-03-29T11:18:05.470 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:05.471 INFO:teuthology.orchestra.run.smithi176.stdout:Total 1.3 MB/s | 964 kB 00:00 2024-03-29T11:18:05.471 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-03-29T11:18:05.493 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-03-29T11:18:05.493 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-03-29T11:18:05.569 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-03-29T11:18:05.570 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-03-29T11:18:05.752 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-03-29T11:18:05.886 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-29T11:18:05.953 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:18:06.694 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:18:06.694 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-29T11:18:07.017 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-29T11:18:07.017 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T11:18:07.017 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-03-29T11:18:07.017 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-29T11:18:07.017 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T11:18:07.017 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-29T11:18:07.354 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph-volume 2024-03-29T11:18:07.934 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:02:18 ago on Fri 29 Mar 2024 11:15:49 AM UTC. 2024-03-29T11:18:08.085 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-29T11:18:08.085 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-29T11:18:08.085 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repository Size 2024-03-29T11:18:08.086 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-29T11:18:08.086 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-03-29T11:18:08.086 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-29T11:18:08.086 INFO:teuthology.orchestra.run.smithi176.stdout:Installing dependencies: 2024-03-29T11:18:08.086 INFO:teuthology.orchestra.run.smithi176.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-29T11:18:08.086 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T11:18:08.086 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-03-29T11:18:08.086 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-29T11:18:08.086 INFO:teuthology.orchestra.run.smithi176.stdout:Install 2 Packages 2024-03-29T11:18:08.086 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T11:18:08.087 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 332 k 2024-03-29T11:18:08.087 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 1.5 M 2024-03-29T11:18:08.087 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-03-29T11:18:08.190 INFO:teuthology.orchestra.run.smithi176.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 666 kB/s | 69 kB 00:00 2024-03-29T11:18:08.490 INFO:teuthology.orchestra.run.smithi176.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 653 kB/s | 264 kB 00:00 2024-03-29T11:18:08.492 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:08.492 INFO:teuthology.orchestra.run.smithi176.stdout:Total 819 kB/s | 332 kB 00:00 2024-03-29T11:18:08.492 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-03-29T11:18:08.501 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-03-29T11:18:08.501 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-03-29T11:18:08.539 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-03-29T11:18:08.539 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-03-29T11:18:08.686 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-03-29T11:18:08.863 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-29T11:18:08.893 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:18:08.909 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:18:08.909 INFO:teuthology.orchestra.run.smithi176.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:18:08.909 INFO:teuthology.orchestra.run.smithi176.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-29T11:18:08.910 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T11:18:09.654 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-29T11:18:09.997 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-29T11:18:09.997 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T11:18:09.997 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-03-29T11:18:09.997 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:18:09.997 INFO:teuthology.orchestra.run.smithi176.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-29T11:18:09.997 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T11:18:09.998 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-29T11:18:10.333 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install librados-devel 2024-03-29T11:18:10.909 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:02:21 ago on Fri 29 Mar 2024 11:15:49 AM UTC. 2024-03-29T11:18:11.061 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-29T11:18:11.062 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-29T11:18:11.062 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repo Size 2024-03-29T11:18:11.062 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-29T11:18:11.062 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-03-29T11:18:11.063 INFO:teuthology.orchestra.run.smithi176.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-29T11:18:11.063 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T11:18:11.063 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-03-29T11:18:11.063 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-29T11:18:11.063 INFO:teuthology.orchestra.run.smithi176.stdout:Install 1 Package 2024-03-29T11:18:11.063 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T11:18:11.063 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 127 k 2024-03-29T11:18:11.063 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 456 k 2024-03-29T11:18:11.063 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-03-29T11:18:11.623 INFO:teuthology.orchestra.run.smithi176.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 226 kB/s | 127 kB 00:00 2024-03-29T11:18:11.623 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:11.624 INFO:teuthology.orchestra.run.smithi176.stdout:Total 225 kB/s | 127 kB 00:00 2024-03-29T11:18:11.624 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-03-29T11:18:11.629 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-03-29T11:18:11.630 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-03-29T11:18:11.686 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-03-29T11:18:11.686 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-03-29T11:18:11.831 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-03-29T11:18:11.963 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:12.499 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:12.771 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:12.772 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T11:18:12.772 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-03-29T11:18:12.772 INFO:teuthology.orchestra.run.smithi176.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:12.772 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T11:18:12.772 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-29T11:18:13.055 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install libcephfs2 2024-03-29T11:18:13.625 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:02:24 ago on Fri 29 Mar 2024 11:15:49 AM UTC. 2024-03-29T11:18:13.727 INFO:teuthology.orchestra.run.smithi176.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:18:13.780 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-29T11:18:13.781 INFO:teuthology.orchestra.run.smithi176.stdout:Nothing to do. 2024-03-29T11:18:13.782 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-29T11:18:13.856 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install libcephfs-devel 2024-03-29T11:18:14.391 INFO:teuthology.orchestra.run.smithi114.stdout:Extra Packages for Enterprise Linux 163 kB/s | 21 MB 02:09 2024-03-29T11:18:14.422 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:02:25 ago on Fri 29 Mar 2024 11:15:49 AM UTC. 2024-03-29T11:18:14.574 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-29T11:18:14.574 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-29T11:18:14.574 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repo Size 2024-03-29T11:18:14.575 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-29T11:18:14.575 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-03-29T11:18:14.575 INFO:teuthology.orchestra.run.smithi176.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-29T11:18:14.575 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T11:18:14.575 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-03-29T11:18:14.575 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-29T11:18:14.575 INFO:teuthology.orchestra.run.smithi176.stdout:Install 1 Package 2024-03-29T11:18:14.575 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T11:18:14.575 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 31 k 2024-03-29T11:18:14.575 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 140 k 2024-03-29T11:18:14.575 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-03-29T11:18:14.842 INFO:teuthology.orchestra.run.smithi176.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 116 kB/s | 31 kB 00:00 2024-03-29T11:18:14.843 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:14.843 INFO:teuthology.orchestra.run.smithi176.stdout:Total 116 kB/s | 31 kB 00:00 2024-03-29T11:18:14.844 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-03-29T11:18:14.847 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-03-29T11:18:14.847 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-03-29T11:18:14.876 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-03-29T11:18:14.876 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-03-29T11:18:15.014 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-03-29T11:18:15.108 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:15.476 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:15.778 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:15.779 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T11:18:15.779 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-03-29T11:18:15.779 INFO:teuthology.orchestra.run.smithi176.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:15.779 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T11:18:15.779 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-29T11:18:15.961 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install librados2 2024-03-29T11:18:16.525 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:02:27 ago on Fri 29 Mar 2024 11:15:49 AM UTC. 2024-03-29T11:18:16.621 INFO:teuthology.orchestra.run.smithi176.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:18:16.672 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-29T11:18:16.673 INFO:teuthology.orchestra.run.smithi176.stdout:Nothing to do. 2024-03-29T11:18:16.674 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-29T11:18:16.723 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install librbd1 2024-03-29T11:18:17.287 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:02:28 ago on Fri 29 Mar 2024 11:15:49 AM UTC. 2024-03-29T11:18:17.385 INFO:teuthology.orchestra.run.smithi176.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:18:17.438 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-29T11:18:17.439 INFO:teuthology.orchestra.run.smithi176.stdout:Nothing to do. 2024-03-29T11:18:17.440 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-29T11:18:17.488 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install python3-rados 2024-03-29T11:18:18.051 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:02:29 ago on Fri 29 Mar 2024 11:15:49 AM UTC. 2024-03-29T11:18:18.148 INFO:teuthology.orchestra.run.smithi176.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:18:18.200 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-29T11:18:18.201 INFO:teuthology.orchestra.run.smithi176.stdout:Nothing to do. 2024-03-29T11:18:18.201 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-29T11:18:18.245 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install python3-rgw 2024-03-29T11:18:18.809 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:02:29 ago on Fri 29 Mar 2024 11:15:49 AM UTC. 2024-03-29T11:18:18.907 INFO:teuthology.orchestra.run.smithi176.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:18:18.959 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-29T11:18:18.961 INFO:teuthology.orchestra.run.smithi176.stdout:Nothing to do. 2024-03-29T11:18:18.961 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-29T11:18:19.016 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install python3-cephfs 2024-03-29T11:18:19.332 INFO:teuthology.orchestra.run.smithi114.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-03-29T11:18:19.585 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:02:30 ago on Fri 29 Mar 2024 11:15:49 AM UTC. 2024-03-29T11:18:19.684 INFO:teuthology.orchestra.run.smithi176.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:18:19.736 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-29T11:18:19.737 INFO:teuthology.orchestra.run.smithi176.stdout:Nothing to do. 2024-03-29T11:18:19.737 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-29T11:18:19.793 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install python3-rbd 2024-03-29T11:18:20.358 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:02:31 ago on Fri 29 Mar 2024 11:15:49 AM UTC. 2024-03-29T11:18:20.458 INFO:teuthology.orchestra.run.smithi176.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:18:20.510 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-29T11:18:20.511 INFO:teuthology.orchestra.run.smithi176.stdout:Nothing to do. 2024-03-29T11:18:20.511 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-29T11:18:20.563 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install rbd-fuse 2024-03-29T11:18:21.129 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:02:32 ago on Fri 29 Mar 2024 11:15:49 AM UTC. 2024-03-29T11:18:21.277 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-29T11:18:21.278 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-29T11:18:21.278 INFO:teuthology.orchestra.run.smithi176.stdout: Package Architecture Version Repository Size 2024-03-29T11:18:21.278 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-29T11:18:21.278 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-03-29T11:18:21.278 INFO:teuthology.orchestra.run.smithi176.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-29T11:18:21.278 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T11:18:21.278 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-03-29T11:18:21.278 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-29T11:18:21.278 INFO:teuthology.orchestra.run.smithi176.stdout:Install 1 Package 2024-03-29T11:18:21.278 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T11:18:21.279 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 85 k 2024-03-29T11:18:21.279 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 231 k 2024-03-29T11:18:21.279 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-03-29T11:18:21.496 INFO:teuthology.orchestra.run.smithi176.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 392 kB/s | 85 kB 00:00 2024-03-29T11:18:21.496 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:21.496 INFO:teuthology.orchestra.run.smithi176.stdout:Total 390 kB/s | 85 kB 00:00 2024-03-29T11:18:21.497 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-03-29T11:18:21.504 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-03-29T11:18:21.504 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-03-29T11:18:21.559 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-03-29T11:18:21.559 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-03-29T11:18:21.697 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-03-29T11:18:21.810 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:21.916 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-03-29T11:18:21.928 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-29T11:18:21.928 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repository Size 2024-03-29T11:18:21.928 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-29T11:18:21.928 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-03-29T11:18:21.928 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-29T11:18:21.928 INFO:teuthology.orchestra.run.smithi114.stdout:Upgrading: 2024-03-29T11:18:21.928 INFO:teuthology.orchestra.run.smithi114.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-29T11:18:21.928 INFO:teuthology.orchestra.run.smithi114.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:18:21.928 INFO:teuthology.orchestra.run.smithi114.stdout:Installing dependencies: 2024-03-29T11:18:21.928 INFO:teuthology.orchestra.run.smithi114.stdout: boost-program-options 2024-03-29T11:18:21.928 INFO:teuthology.orchestra.run.smithi114.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-29T11:18:21.928 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-29T11:18:21.929 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-29T11:18:21.929 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-29T11:18:21.929 INFO:teuthology.orchestra.run.smithi114.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-29T11:18:21.929 INFO:teuthology.orchestra.run.smithi114.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-29T11:18:21.929 INFO:teuthology.orchestra.run.smithi114.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-29T11:18:21.929 INFO:teuthology.orchestra.run.smithi114.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-29T11:18:21.929 INFO:teuthology.orchestra.run.smithi114.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-29T11:18:21.929 INFO:teuthology.orchestra.run.smithi114.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-29T11:18:21.929 INFO:teuthology.orchestra.run.smithi114.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-29T11:18:21.929 INFO:teuthology.orchestra.run.smithi114.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-29T11:18:21.929 INFO:teuthology.orchestra.run.smithi114.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-29T11:18:21.929 INFO:teuthology.orchestra.run.smithi114.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-29T11:18:21.929 INFO:teuthology.orchestra.run.smithi114.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-29T11:18:21.929 INFO:teuthology.orchestra.run.smithi114.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-29T11:18:21.929 INFO:teuthology.orchestra.run.smithi114.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-29T11:18:21.930 INFO:teuthology.orchestra.run.smithi114.stdout: python3-ceph-argparse 2024-03-29T11:18:21.930 INFO:teuthology.orchestra.run.smithi114.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-29T11:18:21.930 INFO:teuthology.orchestra.run.smithi114.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-29T11:18:21.930 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-29T11:18:21.930 INFO:teuthology.orchestra.run.smithi114.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-29T11:18:21.930 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-29T11:18:21.930 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-29T11:18:21.930 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-29T11:18:21.930 INFO:teuthology.orchestra.run.smithi114.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-03-29T11:18:21.930 INFO:teuthology.orchestra.run.smithi114.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-03-29T11:18:21.930 INFO:teuthology.orchestra.run.smithi114.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-29T11:18:21.930 INFO:teuthology.orchestra.run.smithi114.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-29T11:18:21.930 INFO:teuthology.orchestra.run.smithi114.stdout:Installing weak dependencies: 2024-03-29T11:18:21.931 INFO:teuthology.orchestra.run.smithi114.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-03-29T11:18:21.931 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T11:18:21.931 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-03-29T11:18:21.931 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-29T11:18:21.931 INFO:teuthology.orchestra.run.smithi114.stdout:Install 30 Packages 2024-03-29T11:18:21.931 INFO:teuthology.orchestra.run.smithi114.stdout:Upgrade 2 Packages 2024-03-29T11:18:21.931 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T11:18:21.931 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 58 M 2024-03-29T11:18:21.931 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-03-29T11:18:22.227 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:22.508 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:22.508 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T11:18:22.508 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-03-29T11:18:22.508 INFO:teuthology.orchestra.run.smithi176.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:22.508 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T11:18:22.509 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-29T11:18:22.724 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install rbd-mirror 2024-03-29T11:18:22.891 INFO:teuthology.orchestra.run.smithi114.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 9.2 MB/s | 5.3 MB 00:00 2024-03-29T11:18:22.941 INFO:teuthology.orchestra.run.smithi114.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 506 kB/s | 25 kB 00:00 2024-03-29T11:18:23.025 INFO:teuthology.orchestra.run.smithi114.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 8.7 MB/s | 742 kB 00:00 2024-03-29T11:18:23.092 INFO:teuthology.orchestra.run.smithi114.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 7.3 MB/s | 498 kB 00:00 2024-03-29T11:18:23.299 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:02:34 ago on Fri 29 Mar 2024 11:15:49 AM UTC. 2024-03-29T11:18:23.455 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-29T11:18:23.456 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-29T11:18:23.456 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repo Size 2024-03-29T11:18:23.456 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-29T11:18:23.456 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-03-29T11:18:23.456 INFO:teuthology.orchestra.run.smithi176.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:18:23.456 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T11:18:23.456 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-03-29T11:18:23.456 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-29T11:18:23.456 INFO:teuthology.orchestra.run.smithi176.stdout:Install 1 Package 2024-03-29T11:18:23.456 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T11:18:23.456 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 3.1 M 2024-03-29T11:18:23.456 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 13 M 2024-03-29T11:18:23.457 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-03-29T11:18:23.475 INFO:teuthology.orchestra.run.smithi114.stdout:(5/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_6 13 MB/s | 5.1 MB 00:00 2024-03-29T11:18:23.534 INFO:teuthology.orchestra.run.smithi114.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 776 kB/s | 45 kB 00:00 2024-03-29T11:18:23.609 INFO:teuthology.orchestra.run.smithi114.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 1.7 MB/s | 131 kB 00:00 2024-03-29T11:18:23.776 INFO:teuthology.orchestra.run.smithi114.stdout:(8/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 6.6 MB/s | 9.7 MB 00:01 2024-03-29T11:18:23.801 INFO:teuthology.orchestra.run.smithi114.stdout:(9/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 843 kB/s | 162 kB 00:00 2024-03-29T11:18:23.843 INFO:teuthology.orchestra.run.smithi114.stdout:(10/32): python3-rados-19.0.0-1439.g5dcb190d.el 4.7 MB/s | 320 kB 00:00 2024-03-29T11:18:23.868 INFO:teuthology.orchestra.run.smithi114.stdout:(11/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 4.3 MB/s | 298 kB 00:00 2024-03-29T11:18:23.927 INFO:teuthology.orchestra.run.smithi114.stdout:(12/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 1.2 MB/s | 99 kB 00:00 2024-03-29T11:18:23.986 INFO:teuthology.orchestra.run.smithi114.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 906 kB/s | 106 kB 00:00 2024-03-29T11:18:24.011 INFO:teuthology.orchestra.run.smithi114.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.9 MB/s | 159 kB 00:00 2024-03-29T11:18:24.036 INFO:teuthology.orchestra.run.smithi114.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 882 kB/s | 44 kB 00:00 2024-03-29T11:18:24.078 INFO:teuthology.orchestra.run.smithi114.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 9.7 MB/s | 661 kB 00:00 2024-03-29T11:18:24.112 INFO:teuthology.orchestra.run.smithi114.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.7 MB/s | 282 kB 00:00 2024-03-29T11:18:24.137 INFO:teuthology.orchestra.run.smithi114.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 706 kB/s | 41 kB 00:00 2024-03-29T11:18:24.162 INFO:teuthology.orchestra.run.smithi114.stdout:(19/32): qatlib-24.02.0-1.el9.x86_64.rpm 4.3 MB/s | 220 kB 00:00 2024-03-29T11:18:24.188 INFO:teuthology.orchestra.run.smithi114.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 696 kB/s | 35 kB 00:00 2024-03-29T11:18:24.213 INFO:teuthology.orchestra.run.smithi114.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 922 kB/s | 46 kB 00:00 2024-03-29T11:18:24.588 INFO:teuthology.orchestra.run.smithi114.stdout:(22/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 772 kB/s | 309 kB 00:00 2024-03-29T11:18:24.664 INFO:teuthology.orchestra.run.smithi114.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 347 kB/s | 26 kB 00:00 2024-03-29T11:18:24.690 INFO:teuthology.orchestra.run.smithi176.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 2.5 MB/s | 3.1 MB 00:01 2024-03-29T11:18:24.691 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:24.691 INFO:teuthology.orchestra.run.smithi176.stdout:Total 2.5 MB/s | 3.1 MB 00:01 2024-03-29T11:18:24.691 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-03-29T11:18:24.701 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-03-29T11:18:24.701 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-03-29T11:18:24.739 INFO:teuthology.orchestra.run.smithi114.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 652 kB/s | 49 kB 00:00 2024-03-29T11:18:24.777 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-03-29T11:18:24.777 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-03-29T11:18:24.931 INFO:teuthology.orchestra.run.smithi114.stdout:(25/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 7.6 MB/s | 20 MB 00:02 2024-03-29T11:18:24.956 INFO:teuthology.orchestra.run.smithi114.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 313 kB/s | 67 kB 00:00 2024-03-29T11:18:25.056 INFO:teuthology.orchestra.run.smithi114.stdout:(27/32): libarrow-9.0.0-10.el9.x86_64.rpm 5.2 MB/s | 4.4 MB 00:00 2024-03-29T11:18:25.165 INFO:teuthology.orchestra.run.smithi114.stdout:(28/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.5 MB/s | 838 kB 00:00 2024-03-29T11:18:25.190 INFO:teuthology.orchestra.run.smithi114.stdout:(29/32): re2-20211101-3.el9.x86_64.rpm 823 kB/s | 192 kB 00:00 2024-03-29T11:18:25.205 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-03-29T11:18:25.232 INFO:teuthology.orchestra.run.smithi114.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 9.1 MB/s | 1.6 MB 00:00 2024-03-29T11:18:25.257 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:25.282 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:25.282 INFO:teuthology.orchestra.run.smithi176.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:18:25.282 INFO:teuthology.orchestra.run.smithi176.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-29T11:18:25.282 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:18:25.282 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:18:25.282 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T11:18:25.624 INFO:teuthology.orchestra.run.smithi114.stdout:(31/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 7.2 MB/s | 3.1 MB 00:00 2024-03-29T11:18:25.699 INFO:teuthology.orchestra.run.smithi114.stdout:(32/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 6.3 MB/s | 3.4 MB 00:00 2024-03-29T11:18:25.705 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:25.705 INFO:teuthology.orchestra.run.smithi114.stdout:Total 15 MB/s | 58 MB 00:03 2024-03-29T11:18:25.705 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-03-29T11:18:25.743 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-03-29T11:18:25.743 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-03-29T11:18:26.218 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-03-29T11:18:26.218 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-03-29T11:18:26.454 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:26.454 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T11:18:26.454 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-03-29T11:18:26.454 INFO:teuthology.orchestra.run.smithi176.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:26.454 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T11:18:26.454 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-29T11:18:26.731 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install rbd-nbd 2024-03-29T11:18:27.312 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:02:38 ago on Fri 29 Mar 2024 11:15:49 AM UTC. 2024-03-29T11:18:27.460 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-29T11:18:27.460 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-29T11:18:27.460 INFO:teuthology.orchestra.run.smithi176.stdout: Package Architecture Version Repository Size 2024-03-29T11:18:27.460 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-29T11:18:27.460 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-03-29T11:18:27.460 INFO:teuthology.orchestra.run.smithi176.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-29T11:18:27.460 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T11:18:27.461 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-03-29T11:18:27.461 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-29T11:18:27.461 INFO:teuthology.orchestra.run.smithi176.stdout:Install 1 Package 2024-03-29T11:18:27.461 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T11:18:27.461 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 171 k 2024-03-29T11:18:27.462 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 494 k 2024-03-29T11:18:27.462 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-03-29T11:18:27.673 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-03-29T11:18:27.756 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-29T11:18:27.795 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-29T11:18:27.820 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-29T11:18:27.852 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:18:27.869 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:18:27.944 INFO:teuthology.orchestra.run.smithi176.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 355 kB/s | 171 kB 00:00 2024-03-29T11:18:27.944 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:27.945 INFO:teuthology.orchestra.run.smithi176.stdout:Total 354 kB/s | 171 kB 00:00 2024-03-29T11:18:27.945 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-03-29T11:18:27.953 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-03-29T11:18:27.953 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-03-29T11:18:28.011 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-03-29T11:18:28.011 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-03-29T11:18:28.013 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:18:28.068 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:18:28.151 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-03-29T11:18:28.258 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:28.390 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-29T11:18:28.458 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:18:28.534 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:18:28.623 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-29T11:18:28.651 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:18:28.697 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:18:28.726 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:28.727 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-29T11:18:28.789 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-29T11:18:28.864 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:18:28.887 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:18:28.950 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:18:28.985 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-29T11:18:29.007 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:29.007 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T11:18:29.007 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-03-29T11:18:29.007 INFO:teuthology.orchestra.run.smithi176.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:29.007 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T11:18:29.007 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-29T11:18:29.094 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-29T11:18:29.137 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-29T11:18:29.177 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-29T11:18:29.210 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-29T11:18:29.233 DEBUG:teuthology.parallel:result is None 2024-03-29T11:18:29.238 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-29T11:18:29.271 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-29T11:18:29.605 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-29T11:18:29.634 INFO:teuthology.orchestra.run.smithi114.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:18:29.724 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:18:29.758 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-29T11:18:29.795 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-29T11:18:30.248 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-29T11:18:30.271 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:18:30.341 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:18:30.410 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-29T11:18:30.448 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-29T11:18:32.600 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:18:32.636 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:18:33.185 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:18:33.223 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:18:33.299 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:18:33.299 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-29T11:18:33.299 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-29T11:18:33.299 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T11:18:33.332 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:18:42.632 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:18:42.632 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /sys 2024-03-29T11:18:42.632 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /proc 2024-03-29T11:18:42.632 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /mnt 2024-03-29T11:18:42.632 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /var/tmp 2024-03-29T11:18:42.632 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /home 2024-03-29T11:18:42.632 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /root 2024-03-29T11:18:42.632 INFO:teuthology.orchestra.run.smithi114.stdout:skipping the directory /tmp 2024-03-29T11:18:42.632 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T11:18:43.547 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:18:43.572 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:18:43.572 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:18:43.572 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-29T11:18:43.572 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:18:43.572 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:18:43.572 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T11:18:43.574 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:18:43.675 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:18:43.675 INFO:teuthology.orchestra.run.smithi114.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:18:44.810 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:18:44.810 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-29T11:18:44.810 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-29T11:18:44.810 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-29T11:18:44.810 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-29T11:18:44.811 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-29T11:18:44.811 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-29T11:18:44.811 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:18:44.811 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-29T11:18:44.811 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-29T11:18:44.811 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-29T11:18:44.811 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-29T11:18:44.811 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:18:44.811 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-29T11:18:44.811 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-29T11:18:44.811 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-29T11:18:44.811 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-29T11:18:44.811 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-29T11:18:44.812 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-29T11:18:44.812 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-29T11:18:44.812 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-29T11:18:44.812 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-29T11:18:44.812 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-29T11:18:44.812 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-29T11:18:44.812 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-29T11:18:44.812 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-29T11:18:44.812 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-29T11:18:44.812 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-29T11:18:44.812 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-29T11:18:44.812 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-29T11:18:44.812 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-29T11:18:44.812 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:18:44.812 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-29T11:18:44.813 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-29T11:18:45.844 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:18:45.844 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T11:18:45.844 INFO:teuthology.orchestra.run.smithi114.stdout:Upgraded: 2024-03-29T11:18:45.844 INFO:teuthology.orchestra.run.smithi114.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:45.844 INFO:teuthology.orchestra.run.smithi114.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:45.844 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-03-29T11:18:45.844 INFO:teuthology.orchestra.run.smithi114.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-29T11:18:45.844 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:45.844 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:45.845 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:45.845 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:45.845 INFO:teuthology.orchestra.run.smithi114.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-29T11:18:45.845 INFO:teuthology.orchestra.run.smithi114.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-29T11:18:45.845 INFO:teuthology.orchestra.run.smithi114.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-29T11:18:45.845 INFO:teuthology.orchestra.run.smithi114.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:45.845 INFO:teuthology.orchestra.run.smithi114.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-29T11:18:45.845 INFO:teuthology.orchestra.run.smithi114.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-29T11:18:45.845 INFO:teuthology.orchestra.run.smithi114.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-29T11:18:45.845 INFO:teuthology.orchestra.run.smithi114.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:45.846 INFO:teuthology.orchestra.run.smithi114.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-29T11:18:45.846 INFO:teuthology.orchestra.run.smithi114.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:45.846 INFO:teuthology.orchestra.run.smithi114.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-29T11:18:45.846 INFO:teuthology.orchestra.run.smithi114.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-29T11:18:45.846 INFO:teuthology.orchestra.run.smithi114.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-29T11:18:45.846 INFO:teuthology.orchestra.run.smithi114.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:45.846 INFO:teuthology.orchestra.run.smithi114.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:45.846 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:45.846 INFO:teuthology.orchestra.run.smithi114.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-29T11:18:45.847 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:45.847 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:45.847 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:45.847 INFO:teuthology.orchestra.run.smithi114.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-29T11:18:45.847 INFO:teuthology.orchestra.run.smithi114.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-29T11:18:45.847 INFO:teuthology.orchestra.run.smithi114.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-29T11:18:45.847 INFO:teuthology.orchestra.run.smithi114.stdout: re2-1:20211101-3.el9.x86_64 2024-03-29T11:18:45.847 INFO:teuthology.orchestra.run.smithi114.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-29T11:18:45.847 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T11:18:45.848 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-03-29T11:18:46.222 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph-test 2024-03-29T11:18:46.775 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:00:27 ago on Fri 29 Mar 2024 11:18:19 AM UTC. 2024-03-29T11:18:46.928 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-03-29T11:18:46.929 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-29T11:18:46.929 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repository Size 2024-03-29T11:18:46.929 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-29T11:18:46.929 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-03-29T11:18:46.929 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-29T11:18:46.929 INFO:teuthology.orchestra.run.smithi114.stdout:Installing dependencies: 2024-03-29T11:18:46.929 INFO:teuthology.orchestra.run.smithi114.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-29T11:18:46.929 INFO:teuthology.orchestra.run.smithi114.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-29T11:18:46.929 INFO:teuthology.orchestra.run.smithi114.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-29T11:18:46.929 INFO:teuthology.orchestra.run.smithi114.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-29T11:18:46.929 INFO:teuthology.orchestra.run.smithi114.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-29T11:18:46.929 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T11:18:46.929 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-03-29T11:18:46.930 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-29T11:18:46.930 INFO:teuthology.orchestra.run.smithi114.stdout:Install 6 Packages 2024-03-29T11:18:46.930 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T11:18:46.930 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 48 M 2024-03-29T11:18:46.930 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 204 M 2024-03-29T11:18:46.930 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-03-29T11:18:47.365 INFO:teuthology.orchestra.run.smithi114.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.3 MB/s | 217 kB 00:00 2024-03-29T11:18:47.415 INFO:teuthology.orchestra.run.smithi114.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2024-03-29T11:18:47.449 INFO:teuthology.orchestra.run.smithi114.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-03-29T11:18:47.583 INFO:teuthology.orchestra.run.smithi114.stdout:(4/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 429 kB/s | 163 kB 00:00 2024-03-29T11:18:48.299 INFO:teuthology.orchestra.run.smithi114.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 220 kB/s | 187 kB 00:00 2024-03-29T11:18:49.700 INFO:teuthology.orchestra.run.smithi114.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 19 MB/s | 47 MB 00:02 2024-03-29T11:18:49.704 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:49.704 INFO:teuthology.orchestra.run.smithi114.stdout:Total 17 MB/s | 48 MB 00:02 2024-03-29T11:18:49.718 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-03-29T11:18:49.739 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-03-29T11:18:49.739 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-03-29T11:18:50.116 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-03-29T11:18:50.116 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-03-29T11:18:50.689 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-03-29T11:18:50.767 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-29T11:18:50.804 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-29T11:18:50.844 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:18:50.896 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-29T11:18:50.920 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:18:55.370 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:18:55.444 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:18:57.897 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:18:57.897 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-29T11:18:57.897 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-29T11:18:57.897 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:18:57.897 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-29T11:18:57.897 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-29T11:18:58.276 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-29T11:18:58.276 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T11:18:58.277 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-03-29T11:18:58.277 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:58.277 INFO:teuthology.orchestra.run.smithi114.stdout: jq-1.6-16.el9.x86_64 2024-03-29T11:18:58.277 INFO:teuthology.orchestra.run.smithi114.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:58.277 INFO:teuthology.orchestra.run.smithi114.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-29T11:18:58.277 INFO:teuthology.orchestra.run.smithi114.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-29T11:18:58.277 INFO:teuthology.orchestra.run.smithi114.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-29T11:18:58.277 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T11:18:58.277 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-03-29T11:18:58.570 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph 2024-03-29T11:18:59.143 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:00:40 ago on Fri 29 Mar 2024 11:18:19 AM UTC. 2024-03-29T11:18:59.313 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-03-29T11:18:59.316 INFO:teuthology.orchestra.run.smithi114.stdout:========================================================================================= 2024-03-29T11:18:59.316 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repository Size 2024-03-29T11:18:59.316 INFO:teuthology.orchestra.run.smithi114.stdout:========================================================================================= 2024-03-29T11:18:59.316 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-03-29T11:18:59.316 INFO:teuthology.orchestra.run.smithi114.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-29T11:18:59.316 INFO:teuthology.orchestra.run.smithi114.stdout:Installing dependencies: 2024-03-29T11:18:59.316 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-29T11:18:59.317 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-29T11:18:59.317 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-29T11:18:59.317 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-29T11:18:59.317 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-29T11:18:59.317 INFO:teuthology.orchestra.run.smithi114.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-29T11:18:59.317 INFO:teuthology.orchestra.run.smithi114.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-29T11:18:59.317 INFO:teuthology.orchestra.run.smithi114.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-29T11:18:59.317 INFO:teuthology.orchestra.run.smithi114.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-29T11:18:59.317 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-29T11:18:59.317 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-29T11:18:59.317 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-29T11:18:59.317 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-29T11:18:59.317 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-29T11:18:59.317 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-29T11:18:59.317 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-29T11:18:59.318 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-29T11:18:59.318 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-29T11:18:59.318 INFO:teuthology.orchestra.run.smithi114.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-29T11:18:59.318 INFO:teuthology.orchestra.run.smithi114.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-29T11:18:59.318 INFO:teuthology.orchestra.run.smithi114.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-29T11:18:59.318 INFO:teuthology.orchestra.run.smithi114.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-29T11:18:59.318 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-29T11:18:59.318 INFO:teuthology.orchestra.run.smithi114.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-29T11:18:59.318 INFO:teuthology.orchestra.run.smithi114.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-29T11:18:59.318 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-29T11:18:59.318 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-29T11:18:59.318 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-29T11:18:59.318 INFO:teuthology.orchestra.run.smithi114.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-29T11:18:59.319 INFO:teuthology.orchestra.run.smithi114.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-29T11:18:59.319 INFO:teuthology.orchestra.run.smithi114.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-29T11:18:59.319 INFO:teuthology.orchestra.run.smithi114.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-29T11:18:59.319 INFO:teuthology.orchestra.run.smithi114.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-29T11:18:59.319 INFO:teuthology.orchestra.run.smithi114.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-29T11:18:59.319 INFO:teuthology.orchestra.run.smithi114.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-29T11:18:59.319 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T11:18:59.319 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-03-29T11:18:59.319 INFO:teuthology.orchestra.run.smithi114.stdout:========================================================================================= 2024-03-29T11:18:59.319 INFO:teuthology.orchestra.run.smithi114.stdout:Install 36 Packages 2024-03-29T11:18:59.319 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T11:18:59.320 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 30 M 2024-03-29T11:18:59.320 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 107 M 2024-03-29T11:18:59.320 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-03-29T11:18:59.811 INFO:teuthology.orchestra.run.smithi114.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 39 kB/s | 6.6 kB 00:00 2024-03-29T11:19:00.086 INFO:teuthology.orchestra.run.smithi114.stdout:(2/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 4.8 MB/s | 2.1 MB 00:00 2024-03-29T11:19:00.270 INFO:teuthology.orchestra.run.smithi114.stdout:(3/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 2.4 MB/s | 1.5 MB 00:00 2024-03-29T11:19:00.354 INFO:teuthology.orchestra.run.smithi114.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 2.9 MB/s | 247 kB 00:00 2024-03-29T11:19:00.446 INFO:teuthology.orchestra.run.smithi114.stdout:(5/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86_ 7.4 MB/s | 4.7 MB 00:00 2024-03-29T11:19:00.513 INFO:teuthology.orchestra.run.smithi114.stdout:(6/36): lua-5.4.4-4.el9.x86_64.rpm 1.2 MB/s | 187 kB 00:00 2024-03-29T11:19:00.538 INFO:teuthology.orchestra.run.smithi114.stdout:(7/36): python3-mako-1.1.4-6.el9.noarch.rpm 1.6 MB/s | 144 kB 00:00 2024-03-29T11:19:00.555 INFO:teuthology.orchestra.run.smithi114.stdout:(8/36): python3-markupsafe-1.1.1-12.el9.x86_64. 757 kB/s | 32 kB 00:00 2024-03-29T11:19:00.572 INFO:teuthology.orchestra.run.smithi114.stdout:(9/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-29T11:19:00.589 INFO:teuthology.orchestra.run.smithi114.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-29T11:19:00.614 INFO:teuthology.orchestra.run.smithi114.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.6 MB/s | 241 kB 00:00 2024-03-29T11:19:00.673 INFO:teuthology.orchestra.run.smithi114.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 14 MB/s | 1.1 MB 00:00 2024-03-29T11:19:00.698 INFO:teuthology.orchestra.run.smithi114.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 1.2 MB/s | 103 kB 00:00 2024-03-29T11:19:00.723 INFO:teuthology.orchestra.run.smithi114.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-03-29T11:19:00.748 INFO:teuthology.orchestra.run.smithi114.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.6 MB/s | 187 kB 00:00 2024-03-29T11:19:00.807 INFO:teuthology.orchestra.run.smithi114.stdout:(16/36): python3-requests-2.25.1-8.el9.noarch.r 1.5 MB/s | 125 kB 00:00 2024-03-29T11:19:00.841 INFO:teuthology.orchestra.run.smithi114.stdout:(17/36): lua-devel-5.4.4-4.el9.x86_64.rpm 242 kB/s | 22 kB 00:00 2024-03-29T11:19:01.174 INFO:teuthology.orchestra.run.smithi114.stdout:(18/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 15 MB/s | 17 MB 00:01 2024-03-29T11:19:01.417 INFO:teuthology.orchestra.run.smithi114.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 75 kB/s | 43 kB 00:00 2024-03-29T11:19:01.442 INFO:teuthology.orchestra.run.smithi114.stdout:(20/36): luarocks-3.9.2-1.el9.noarch.rpm 238 kB/s | 151 kB 00:00 2024-03-29T11:19:01.559 INFO:teuthology.orchestra.run.smithi114.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 91 kB/s | 11 kB 00:00 2024-03-29T11:19:01.676 INFO:teuthology.orchestra.run.smithi114.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 151 kB/s | 18 kB 00:00 2024-03-29T11:19:01.776 INFO:teuthology.orchestra.run.smithi114.stdout:(23/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 286 kB/s | 172 kB 00:00 2024-03-29T11:19:01.801 INFO:teuthology.orchestra.run.smithi114.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 185 kB/s | 23 kB 00:00 2024-03-29T11:19:01.827 INFO:teuthology.orchestra.run.smithi114.stdout:(25/36): python3-cherrypy-18.6.1-2.el9.noarch.r 875 kB/s | 358 kB 00:00 2024-03-29T11:19:01.893 INFO:teuthology.orchestra.run.smithi114.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 166 kB/s | 19 kB 00:00 2024-03-29T11:19:01.919 INFO:teuthology.orchestra.run.smithi114.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 168 kB/s | 20 kB 00:00 2024-03-29T11:19:01.944 INFO:teuthology.orchestra.run.smithi114.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 395 kB/s | 46 kB 00:00 2024-03-29T11:19:02.036 INFO:teuthology.orchestra.run.smithi114.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 555 kB/s | 79 kB 00:00 2024-03-29T11:19:02.061 INFO:teuthology.orchestra.run.smithi114.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 140 kB/s | 16 kB 00:00 2024-03-29T11:19:02.178 INFO:teuthology.orchestra.run.smithi114.stdout:(31/36): python3-pecan-1.4.2-3.el9.noarch.rpm 1.0 MB/s | 272 kB 00:00 2024-03-29T11:19:02.203 INFO:teuthology.orchestra.run.smithi114.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 252 kB/s | 36 kB 00:00 2024-03-29T11:19:02.229 INFO:teuthology.orchestra.run.smithi114.stdout:(33/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 467 kB/s | 90 kB 00:00 2024-03-29T11:19:02.346 INFO:teuthology.orchestra.run.smithi114.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 1.3 MB/s | 230 kB 00:00 2024-03-29T11:19:02.371 INFO:teuthology.orchestra.run.smithi114.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 141 kB/s | 20 kB 00:00 2024-03-29T11:19:02.404 INFO:teuthology.orchestra.run.smithi114.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.1 MB/s | 427 kB 00:00 2024-03-29T11:19:02.411 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:19:02.411 INFO:teuthology.orchestra.run.smithi114.stdout:Total 9.6 MB/s | 30 MB 00:03 2024-03-29T11:19:02.435 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-03-29T11:19:02.477 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-03-29T11:19:02.478 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-03-29T11:19:02.881 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-03-29T11:19:02.881 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-03-29T11:19:04.240 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-03-29T11:19:04.299 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-29T11:19:04.339 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-29T11:19:04.368 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-29T11:19:04.439 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-29T11:19:04.492 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-29T11:19:04.525 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-29T11:19:04.617 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-29T11:19:04.704 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-29T11:19:04.744 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-29T11:19:04.817 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-29T11:19:04.879 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-29T11:19:04.916 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-29T11:19:04.965 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-29T11:19:05.021 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-29T11:19:05.209 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-29T11:19:05.263 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-29T11:19:05.301 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-29T11:19:05.336 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-29T11:19:05.366 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-29T11:19:05.443 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-29T11:19:05.476 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:19:05.509 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-29T11:19:05.550 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-29T11:19:05.585 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-29T11:19:05.620 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-29T11:19:05.710 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-29T11:19:05.750 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-29T11:19:05.826 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-29T11:19:05.975 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-29T11:19:06.097 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:19:06.253 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-29T11:19:06.282 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:19:06.307 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:19:06.308 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:19:06.308 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-29T11:19:06.308 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:19:06.308 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:19:06.308 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T11:19:07.733 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:19:07.760 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:19:07.760 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:19:07.760 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-29T11:19:07.761 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:19:07.761 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:19:07.761 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T11:19:08.209 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:19:08.232 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:19:08.232 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:19:08.232 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-29T11:19:08.232 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:19:08.232 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:19:08.232 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T11:19:08.451 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:19:08.474 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:19:08.474 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:19:08.474 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-29T11:19:08.474 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:19:08.474 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:19:08.474 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T11:19:08.540 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:19:10.287 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:19:10.287 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-29T11:19:10.287 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-29T11:19:10.287 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-29T11:19:10.288 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-29T11:19:10.288 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-29T11:19:10.288 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-29T11:19:10.288 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-29T11:19:10.288 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-29T11:19:10.288 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-29T11:19:10.288 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-29T11:19:10.288 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-29T11:19:10.288 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-29T11:19:10.288 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-29T11:19:10.288 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-29T11:19:10.288 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-29T11:19:10.288 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-29T11:19:10.289 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-29T11:19:10.289 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-29T11:19:10.289 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-29T11:19:10.289 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-29T11:19:10.289 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:19:10.289 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-29T11:19:10.289 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-29T11:19:10.289 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-29T11:19:10.289 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-29T11:19:10.289 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-29T11:19:10.289 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-29T11:19:10.289 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-29T11:19:10.290 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-29T11:19:10.290 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:19:10.290 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-29T11:19:10.291 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-29T11:19:10.291 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-29T11:19:10.291 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-29T11:19:10.291 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-29T11:19:11.279 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-29T11:19:11.280 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T11:19:11.280 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-03-29T11:19:11.280 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:19:11.280 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:19:11.280 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:19:11.280 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:19:11.280 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:19:11.280 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:19:11.280 INFO:teuthology.orchestra.run.smithi114.stdout: lua-5.4.4-4.el9.x86_64 2024-03-29T11:19:11.280 INFO:teuthology.orchestra.run.smithi114.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-29T11:19:11.280 INFO:teuthology.orchestra.run.smithi114.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-29T11:19:11.281 INFO:teuthology.orchestra.run.smithi114.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-29T11:19:11.281 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-29T11:19:11.281 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-29T11:19:11.281 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-29T11:19:11.281 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-29T11:19:11.281 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-29T11:19:11.281 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-29T11:19:11.281 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-29T11:19:11.281 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-29T11:19:11.281 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-29T11:19:11.281 INFO:teuthology.orchestra.run.smithi114.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-29T11:19:11.281 INFO:teuthology.orchestra.run.smithi114.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-29T11:19:11.282 INFO:teuthology.orchestra.run.smithi114.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-29T11:19:11.282 INFO:teuthology.orchestra.run.smithi114.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-29T11:19:11.282 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-29T11:19:11.282 INFO:teuthology.orchestra.run.smithi114.stdout: python3-ply-3.11-14.el9.noarch 2024-03-29T11:19:11.282 INFO:teuthology.orchestra.run.smithi114.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-29T11:19:11.282 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-29T11:19:11.282 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-29T11:19:11.282 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-29T11:19:11.282 INFO:teuthology.orchestra.run.smithi114.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-29T11:19:11.282 INFO:teuthology.orchestra.run.smithi114.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-29T11:19:11.282 INFO:teuthology.orchestra.run.smithi114.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-29T11:19:11.282 INFO:teuthology.orchestra.run.smithi114.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-29T11:19:11.282 INFO:teuthology.orchestra.run.smithi114.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-29T11:19:11.283 INFO:teuthology.orchestra.run.smithi114.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-29T11:19:11.283 INFO:teuthology.orchestra.run.smithi114.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-29T11:19:11.283 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T11:19:11.283 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-03-29T11:19:11.608 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph-base 2024-03-29T11:19:12.173 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:00:53 ago on Fri 29 Mar 2024 11:18:19 AM UTC. 2024-03-29T11:19:12.270 INFO:teuthology.orchestra.run.smithi114.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:19:12.322 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-03-29T11:19:12.323 INFO:teuthology.orchestra.run.smithi114.stdout:Nothing to do. 2024-03-29T11:19:12.323 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-03-29T11:19:12.396 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install cephadm 2024-03-29T11:19:12.955 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:00:53 ago on Fri 29 Mar 2024 11:18:19 AM UTC. 2024-03-29T11:19:13.107 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-03-29T11:19:13.108 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-29T11:19:13.108 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repository Size 2024-03-29T11:19:13.108 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-29T11:19:13.108 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-03-29T11:19:13.108 INFO:teuthology.orchestra.run.smithi114.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-29T11:19:13.108 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T11:19:13.108 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-03-29T11:19:13.108 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-29T11:19:13.108 INFO:teuthology.orchestra.run.smithi114.stdout:Install 1 Package 2024-03-29T11:19:13.108 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T11:19:13.108 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 747 k 2024-03-29T11:19:13.109 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 754 k 2024-03-29T11:19:13.109 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-03-29T11:19:13.576 INFO:teuthology.orchestra.run.smithi114.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 1.6 MB/s | 747 kB 00:00 2024-03-29T11:19:13.576 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:19:13.576 INFO:teuthology.orchestra.run.smithi114.stdout:Total 1.6 MB/s | 747 kB 00:00 2024-03-29T11:19:13.576 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-03-29T11:19:13.582 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-03-29T11:19:13.582 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-03-29T11:19:13.587 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-03-29T11:19:13.588 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-03-29T11:19:13.661 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-03-29T11:19:14.150 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:19:14.222 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:19:14.621 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:19:14.903 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:19:14.903 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T11:19:14.903 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-03-29T11:19:14.903 INFO:teuthology.orchestra.run.smithi114.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:19:14.903 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T11:19:14.903 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-03-29T11:19:15.089 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph-immutable-object-cache 2024-03-29T11:19:15.658 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:00:56 ago on Fri 29 Mar 2024 11:18:19 AM UTC. 2024-03-29T11:19:15.810 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-03-29T11:19:15.811 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-29T11:19:15.811 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repo Size 2024-03-29T11:19:15.811 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-29T11:19:15.811 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-03-29T11:19:15.811 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-29T11:19:15.811 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T11:19:15.812 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-03-29T11:19:15.812 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-29T11:19:15.812 INFO:teuthology.orchestra.run.smithi114.stdout:Install 1 Package 2024-03-29T11:19:15.812 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T11:19:15.812 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 145 k 2024-03-29T11:19:15.812 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 436 k 2024-03-29T11:19:15.812 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-03-29T11:19:16.040 INFO:teuthology.orchestra.run.smithi114.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 638 kB/s | 145 kB 00:00 2024-03-29T11:19:16.040 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:19:16.041 INFO:teuthology.orchestra.run.smithi114.stdout:Total 634 kB/s | 145 kB 00:00 2024-03-29T11:19:16.041 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-03-29T11:19:16.048 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-03-29T11:19:16.048 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-03-29T11:19:16.101 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-03-29T11:19:16.101 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-03-29T11:19:16.247 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-03-29T11:19:16.299 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:19:16.323 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:19:16.324 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:19:16.324 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-29T11:19:16.324 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T11:19:17.317 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:19:17.318 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T11:19:17.318 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-03-29T11:19:17.318 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:19:17.318 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T11:19:17.318 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-03-29T11:19:17.597 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph-mgr 2024-03-29T11:19:18.169 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:00:59 ago on Fri 29 Mar 2024 11:18:19 AM UTC. 2024-03-29T11:19:18.269 INFO:teuthology.orchestra.run.smithi114.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:19:18.320 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-03-29T11:19:18.321 INFO:teuthology.orchestra.run.smithi114.stdout:Nothing to do. 2024-03-29T11:19:18.321 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-03-29T11:19:18.395 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph-mgr-dashboard 2024-03-29T11:19:18.955 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:00:59 ago on Fri 29 Mar 2024 11:18:19 AM UTC. 2024-03-29T11:19:19.109 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-03-29T11:19:19.110 INFO:teuthology.orchestra.run.smithi114.stdout:====================================================================================== 2024-03-29T11:19:19.110 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repository Size 2024-03-29T11:19:19.110 INFO:teuthology.orchestra.run.smithi114.stdout:====================================================================================== 2024-03-29T11:19:19.110 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-03-29T11:19:19.110 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-29T11:19:19.110 INFO:teuthology.orchestra.run.smithi114.stdout:Installing dependencies: 2024-03-29T11:19:19.110 INFO:teuthology.orchestra.run.smithi114.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-29T11:19:19.111 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-29T11:19:19.111 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-29T11:19:19.111 INFO:teuthology.orchestra.run.smithi114.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-29T11:19:19.111 INFO:teuthology.orchestra.run.smithi114.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-29T11:19:19.111 INFO:teuthology.orchestra.run.smithi114.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-29T11:19:19.111 INFO:teuthology.orchestra.run.smithi114.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-29T11:19:19.111 INFO:teuthology.orchestra.run.smithi114.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-29T11:19:19.111 INFO:teuthology.orchestra.run.smithi114.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-29T11:19:19.112 INFO:teuthology.orchestra.run.smithi114.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-29T11:19:19.112 INFO:teuthology.orchestra.run.smithi114.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-29T11:19:19.112 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T11:19:19.112 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-03-29T11:19:19.112 INFO:teuthology.orchestra.run.smithi114.stdout:====================================================================================== 2024-03-29T11:19:19.113 INFO:teuthology.orchestra.run.smithi114.stdout:Install 12 Packages 2024-03-29T11:19:19.113 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T11:19:19.114 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 6.9 M 2024-03-29T11:19:19.114 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 26 M 2024-03-29T11:19:19.114 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-03-29T11:19:19.985 INFO:teuthology.orchestra.run.smithi114.stdout:(1/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 63 kB/s | 15 kB 00:00 2024-03-29T11:19:20.044 INFO:teuthology.orchestra.run.smithi114.stdout:(2/12): ceph-grafana-dashboards-19.0.0-1439.g5d 104 kB/s | 31 kB 00:00 2024-03-29T11:19:20.153 INFO:teuthology.orchestra.run.smithi114.stdout:(3/12): protobuf-3.14.0-13.el9.x86_64.rpm 6.0 MB/s | 1.0 MB 00:00 2024-03-29T11:19:20.178 INFO:teuthology.orchestra.run.smithi114.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.8 MB/s | 238 kB 00:00 2024-03-29T11:19:20.295 INFO:teuthology.orchestra.run.smithi114.stdout:(5/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb190 3.2 MB/s | 1.8 MB 00:00 2024-03-29T11:19:20.345 INFO:teuthology.orchestra.run.smithi114.stdout:(6/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 3.2 MB/s | 548 kB 00:00 2024-03-29T11:19:20.362 INFO:teuthology.orchestra.run.smithi114.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 294 kB/s | 19 kB 00:00 2024-03-29T11:19:20.438 INFO:teuthology.orchestra.run.smithi114.stdout:(8/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 22 MB/s | 2.0 MB 00:00 2024-03-29T11:19:20.463 INFO:teuthology.orchestra.run.smithi114.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 1.4 MB/s | 144 kB 00:00 2024-03-29T11:19:20.480 INFO:teuthology.orchestra.run.smithi114.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 730 kB/s | 31 kB 00:00 2024-03-29T11:19:20.521 INFO:teuthology.orchestra.run.smithi114.stdout:(11/12): protobuf-compiler-3.14.0-13.el9.x86_64 2.3 MB/s | 863 kB 00:00 2024-03-29T11:19:20.547 INFO:teuthology.orchestra.run.smithi114.stdout:(12/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.2 MB/s | 188 kB 00:00 2024-03-29T11:19:20.550 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:19:20.551 INFO:teuthology.orchestra.run.smithi114.stdout:Total 4.8 MB/s | 6.9 MB 00:01 2024-03-29T11:19:20.569 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-03-29T11:19:20.585 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-03-29T11:19:20.585 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-03-29T11:19:20.767 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-03-29T11:19:20.767 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-03-29T11:19:21.276 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-03-29T11:19:21.347 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-29T11:19:21.383 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-29T11:19:21.438 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-29T11:19:21.468 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-29T11:19:21.576 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-29T11:19:21.701 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-29T11:19:21.788 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-29T11:19:21.831 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-29T11:19:21.860 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:19:21.882 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-29T11:19:22.125 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-29T11:19:22.158 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:19:22.885 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:19:22.885 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-29T11:19:22.885 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-29T11:19:22.885 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-29T11:19:22.885 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-29T11:19:22.885 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-29T11:19:22.885 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-29T11:19:22.885 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-29T11:19:22.885 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-29T11:19:22.885 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:19:22.886 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-29T11:19:22.886 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-29T11:19:23.226 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-29T11:19:23.226 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T11:19:23.226 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-03-29T11:19:23.226 INFO:teuthology.orchestra.run.smithi114.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-29T11:19:23.226 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:19:23.227 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:19:23.227 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:19:23.227 INFO:teuthology.orchestra.run.smithi114.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-29T11:19:23.227 INFO:teuthology.orchestra.run.smithi114.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-29T11:19:23.227 INFO:teuthology.orchestra.run.smithi114.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-29T11:19:23.227 INFO:teuthology.orchestra.run.smithi114.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-29T11:19:23.228 INFO:teuthology.orchestra.run.smithi114.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-29T11:19:23.228 INFO:teuthology.orchestra.run.smithi114.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-29T11:19:23.228 INFO:teuthology.orchestra.run.smithi114.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-29T11:19:23.228 INFO:teuthology.orchestra.run.smithi114.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-29T11:19:23.228 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T11:19:23.228 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-03-29T11:19:23.478 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-29T11:19:24.038 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:05 ago on Fri 29 Mar 2024 11:18:19 AM UTC. 2024-03-29T11:19:24.193 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-03-29T11:19:24.194 INFO:teuthology.orchestra.run.smithi114.stdout:============================================================================================ 2024-03-29T11:19:24.194 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repository Size 2024-03-29T11:19:24.194 INFO:teuthology.orchestra.run.smithi114.stdout:============================================================================================ 2024-03-29T11:19:24.194 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-03-29T11:19:24.194 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-29T11:19:24.194 INFO:teuthology.orchestra.run.smithi114.stdout:Installing dependencies: 2024-03-29T11:19:24.194 INFO:teuthology.orchestra.run.smithi114.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-29T11:19:24.194 INFO:teuthology.orchestra.run.smithi114.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-29T11:19:24.194 INFO:teuthology.orchestra.run.smithi114.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-29T11:19:24.195 INFO:teuthology.orchestra.run.smithi114.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-29T11:19:24.195 INFO:teuthology.orchestra.run.smithi114.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-29T11:19:24.195 INFO:teuthology.orchestra.run.smithi114.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-29T11:19:24.195 INFO:teuthology.orchestra.run.smithi114.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-29T11:19:24.195 INFO:teuthology.orchestra.run.smithi114.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-29T11:19:24.195 INFO:teuthology.orchestra.run.smithi114.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-29T11:19:24.195 INFO:teuthology.orchestra.run.smithi114.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-29T11:19:24.195 INFO:teuthology.orchestra.run.smithi114.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-29T11:19:24.195 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T11:19:24.195 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-03-29T11:19:24.195 INFO:teuthology.orchestra.run.smithi114.stdout:============================================================================================ 2024-03-29T11:19:24.195 INFO:teuthology.orchestra.run.smithi114.stdout:Install 12 Packages 2024-03-29T11:19:24.195 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T11:19:24.196 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 37 M 2024-03-29T11:19:24.196 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 209 M 2024-03-29T11:19:24.196 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-03-29T11:19:24.662 INFO:teuthology.orchestra.run.smithi114.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 238 kB/s | 32 kB 00:00 2024-03-29T11:19:24.695 INFO:teuthology.orchestra.run.smithi114.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 530 kB/s | 17 kB 00:00 2024-03-29T11:19:24.737 INFO:teuthology.orchestra.run.smithi114.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 862 kB/s | 34 kB 00:00 2024-03-29T11:19:24.787 INFO:teuthology.orchestra.run.smithi114.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 12 MB/s | 3.0 MB 00:00 2024-03-29T11:19:24.980 INFO:teuthology.orchestra.run.smithi114.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 20 MB/s | 4.7 MB 00:00 2024-03-29T11:19:25.055 INFO:teuthology.orchestra.run.smithi114.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 18 MB/s | 4.8 MB 00:00 2024-03-29T11:19:25.080 INFO:teuthology.orchestra.run.smithi114.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.1 MB/s | 217 kB 00:00 2024-03-29T11:19:25.426 INFO:teuthology.orchestra.run.smithi114.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 42 MB/s | 16 MB 00:00 2024-03-29T11:19:25.539 INFO:teuthology.orchestra.run.smithi114.stdout:(9/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.7 MB/s | 803 kB 00:00 2024-03-29T11:19:25.622 INFO:teuthology.orchestra.run.smithi114.stdout:(10/12): python3-devel-3.9.17-2.el9.x86_64.rpm 2.9 MB/s | 245 kB 00:00 2024-03-29T11:19:25.648 INFO:teuthology.orchestra.run.smithi114.stdout:(11/12): libquadmath-11.4.1-3.el9.x86_64.rpm 882 kB/s | 195 kB 00:00 2024-03-29T11:19:25.865 INFO:teuthology.orchestra.run.smithi114.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 5.5 MB/s | 7.4 MB 00:01 2024-03-29T11:19:25.867 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:19:25.867 INFO:teuthology.orchestra.run.smithi114.stdout:Total 22 MB/s | 37 MB 00:01 2024-03-29T11:19:25.907 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-03-29T11:19:25.931 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-03-29T11:19:25.931 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-03-29T11:19:26.302 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-03-29T11:19:26.303 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-03-29T11:19:27.017 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-03-29T11:19:27.096 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-29T11:19:27.135 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-29T11:19:27.227 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:19:27.272 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-29T11:19:27.448 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:19:27.479 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:19:27.562 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-29T11:19:28.237 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-29T11:19:28.350 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-29T11:19:29.607 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-29T11:19:30.634 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-29T11:19:30.670 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:19:33.237 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:19:33.237 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-29T11:19:33.237 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-29T11:19:33.237 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:19:33.237 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-29T11:19:33.237 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:19:33.237 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:19:33.237 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-29T11:19:33.237 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-29T11:19:33.237 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-29T11:19:33.238 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-29T11:19:33.238 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-29T11:19:33.641 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-29T11:19:33.641 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T11:19:33.641 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-03-29T11:19:33.641 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:19:33.641 INFO:teuthology.orchestra.run.smithi114.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-29T11:19:33.641 INFO:teuthology.orchestra.run.smithi114.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-29T11:19:33.641 INFO:teuthology.orchestra.run.smithi114.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-29T11:19:33.642 INFO:teuthology.orchestra.run.smithi114.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-29T11:19:33.642 INFO:teuthology.orchestra.run.smithi114.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-29T11:19:33.642 INFO:teuthology.orchestra.run.smithi114.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-29T11:19:33.642 INFO:teuthology.orchestra.run.smithi114.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-29T11:19:33.642 INFO:teuthology.orchestra.run.smithi114.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-29T11:19:33.642 INFO:teuthology.orchestra.run.smithi114.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-29T11:19:33.642 INFO:teuthology.orchestra.run.smithi114.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-29T11:19:33.642 INFO:teuthology.orchestra.run.smithi114.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-29T11:19:33.642 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T11:19:33.642 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-03-29T11:19:34.022 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph-mgr-rook 2024-03-29T11:19:34.597 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:15 ago on Fri 29 Mar 2024 11:18:19 AM UTC. 2024-03-29T11:19:34.751 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-03-29T11:19:34.752 INFO:teuthology.orchestra.run.smithi114.stdout:======================================================================================== 2024-03-29T11:19:34.753 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repository Size 2024-03-29T11:19:34.753 INFO:teuthology.orchestra.run.smithi114.stdout:======================================================================================== 2024-03-29T11:19:34.753 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-03-29T11:19:34.753 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-29T11:19:34.753 INFO:teuthology.orchestra.run.smithi114.stdout:Installing dependencies: 2024-03-29T11:19:34.753 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-29T11:19:34.753 INFO:teuthology.orchestra.run.smithi114.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-29T11:19:34.753 INFO:teuthology.orchestra.run.smithi114.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-29T11:19:34.753 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-29T11:19:34.753 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-29T11:19:34.753 INFO:teuthology.orchestra.run.smithi114.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-29T11:19:34.753 INFO:teuthology.orchestra.run.smithi114.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-29T11:19:34.754 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-29T11:19:34.754 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-29T11:19:34.754 INFO:teuthology.orchestra.run.smithi114.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-29T11:19:34.754 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-29T11:19:34.754 INFO:teuthology.orchestra.run.smithi114.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-29T11:19:34.754 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T11:19:34.754 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-03-29T11:19:34.754 INFO:teuthology.orchestra.run.smithi114.stdout:======================================================================================== 2024-03-29T11:19:34.754 INFO:teuthology.orchestra.run.smithi114.stdout:Install 13 Packages 2024-03-29T11:19:34.754 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T11:19:34.755 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 2.0 M 2024-03-29T11:19:34.755 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 26 M 2024-03-29T11:19:34.755 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-03-29T11:19:35.046 INFO:teuthology.orchestra.run.smithi114.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 181 kB/s | 24 kB 00:00 2024-03-29T11:19:35.063 INFO:teuthology.orchestra.run.smithi114.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 119 kB/s | 18 kB 00:00 2024-03-29T11:19:35.132 INFO:teuthology.orchestra.run.smithi114.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.6 MB/s | 174 kB 00:00 2024-03-29T11:19:35.155 INFO:teuthology.orchestra.run.smithi114.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.4 MB/s | 132 kB 00:00 2024-03-29T11:19:35.180 INFO:teuthology.orchestra.run.smithi114.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-29T11:19:35.197 INFO:teuthology.orchestra.run.smithi114.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 43 kB 00:00 2024-03-29T11:19:35.214 INFO:teuthology.orchestra.run.smithi114.stdout:(7/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 172 kB/s | 52 kB 00:00 2024-03-29T11:19:35.373 INFO:teuthology.orchestra.run.smithi114.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 85 kB/s | 15 kB 00:00 2024-03-29T11:19:35.390 INFO:teuthology.orchestra.run.smithi114.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 154 kB/s | 32 kB 00:00 2024-03-29T11:19:35.515 INFO:teuthology.orchestra.run.smithi114.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 472 kB/s | 59 kB 00:00 2024-03-29T11:19:35.549 INFO:teuthology.orchestra.run.smithi114.stdout:(11/13): python3-google-auth-2.28.2-1.el9.noarc 606 kB/s | 203 kB 00:00 2024-03-29T11:19:35.690 INFO:teuthology.orchestra.run.smithi114.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 512 kB/s | 90 kB 00:00 2024-03-29T11:19:35.957 INFO:teuthology.orchestra.run.smithi114.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.7 MB/s | 1.0 MB 00:00 2024-03-29T11:19:35.959 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:19:35.959 INFO:teuthology.orchestra.run.smithi114.stdout:Total 1.7 MB/s | 2.0 MB 00:01 2024-03-29T11:19:35.959 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-03-29T11:19:35.974 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-03-29T11:19:35.974 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-03-29T11:19:36.067 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-03-29T11:19:36.068 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-03-29T11:19:36.584 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-03-29T11:19:36.764 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-29T11:19:36.825 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-29T11:19:36.880 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-29T11:19:36.917 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-29T11:19:36.959 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-29T11:19:37.061 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-29T11:19:37.169 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-29T11:19:37.219 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-29T11:19:37.945 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-29T11:19:37.999 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-29T11:19:38.033 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-29T11:19:38.090 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-29T11:19:38.119 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:19:38.951 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:19:38.951 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-29T11:19:38.951 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-29T11:19:38.951 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-29T11:19:38.951 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-29T11:19:38.951 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-29T11:19:38.951 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-29T11:19:38.951 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-29T11:19:38.951 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-29T11:19:38.951 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-29T11:19:38.951 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-29T11:19:38.952 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-29T11:19:38.952 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-29T11:19:39.315 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-29T11:19:39.315 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T11:19:39.315 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-03-29T11:19:39.316 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:19:39.316 INFO:teuthology.orchestra.run.smithi114.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-29T11:19:39.316 INFO:teuthology.orchestra.run.smithi114.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-29T11:19:39.316 INFO:teuthology.orchestra.run.smithi114.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-29T11:19:39.316 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-29T11:19:39.316 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-29T11:19:39.316 INFO:teuthology.orchestra.run.smithi114.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-29T11:19:39.316 INFO:teuthology.orchestra.run.smithi114.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-29T11:19:39.316 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-29T11:19:39.316 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-29T11:19:39.316 INFO:teuthology.orchestra.run.smithi114.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-29T11:19:39.316 INFO:teuthology.orchestra.run.smithi114.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-29T11:19:39.316 INFO:teuthology.orchestra.run.smithi114.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-29T11:19:39.316 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T11:19:39.317 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-03-29T11:19:39.607 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph-mgr-cephadm 2024-03-29T11:19:40.178 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:21 ago on Fri 29 Mar 2024 11:18:19 AM UTC. 2024-03-29T11:19:40.331 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-03-29T11:19:40.331 INFO:teuthology.orchestra.run.smithi114.stdout:======================================================================================== 2024-03-29T11:19:40.331 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repository Size 2024-03-29T11:19:40.331 INFO:teuthology.orchestra.run.smithi114.stdout:======================================================================================== 2024-03-29T11:19:40.331 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-03-29T11:19:40.332 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-29T11:19:40.332 INFO:teuthology.orchestra.run.smithi114.stdout:Installing dependencies: 2024-03-29T11:19:40.332 INFO:teuthology.orchestra.run.smithi114.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-29T11:19:40.332 INFO:teuthology.orchestra.run.smithi114.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-29T11:19:40.332 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-29T11:19:40.332 INFO:teuthology.orchestra.run.smithi114.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-29T11:19:40.332 INFO:teuthology.orchestra.run.smithi114.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-29T11:19:40.332 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T11:19:40.332 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-03-29T11:19:40.332 INFO:teuthology.orchestra.run.smithi114.stdout:======================================================================================== 2024-03-29T11:19:40.332 INFO:teuthology.orchestra.run.smithi114.stdout:Install 6 Packages 2024-03-29T11:19:40.332 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T11:19:40.333 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 6.8 M 2024-03-29T11:19:40.333 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 33 M 2024-03-29T11:19:40.333 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-03-29T11:19:40.968 INFO:teuthology.orchestra.run.smithi114.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 21 MB/s | 5.8 MB 00:00 2024-03-29T11:19:40.993 INFO:teuthology.orchestra.run.smithi114.stdout:(2/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 809 kB/s | 248 kB 00:00 2024-03-29T11:19:41.068 INFO:teuthology.orchestra.run.smithi114.stdout:(3/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 361 kB/s | 138 kB 00:00 2024-03-29T11:19:41.468 INFO:teuthology.orchestra.run.smithi114.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 121 kB/s | 58 kB 00:00 2024-03-29T11:19:41.535 INFO:teuthology.orchestra.run.smithi114.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 121 kB/s | 51 kB 00:00 2024-03-29T11:19:41.685 INFO:teuthology.orchestra.run.smithi114.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 764 kB/s | 548 kB 00:00 2024-03-29T11:19:41.688 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:19:41.688 INFO:teuthology.orchestra.run.smithi114.stdout:Total 5.0 MB/s | 6.8 MB 00:01 2024-03-29T11:19:41.701 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-03-29T11:19:41.710 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-03-29T11:19:41.711 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-03-29T11:19:41.799 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-03-29T11:19:41.800 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-03-29T11:19:42.108 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-03-29T11:19:42.284 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-29T11:19:42.329 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-29T11:19:42.858 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-29T11:19:42.940 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-29T11:19:43.002 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-29T11:19:43.024 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:19:43.950 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:19:43.950 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-29T11:19:43.950 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-29T11:19:43.950 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-29T11:19:43.950 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-29T11:19:43.950 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-29T11:19:44.247 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-29T11:19:44.247 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T11:19:44.247 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-03-29T11:19:44.247 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:19:44.247 INFO:teuthology.orchestra.run.smithi114.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-29T11:19:44.247 INFO:teuthology.orchestra.run.smithi114.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-29T11:19:44.247 INFO:teuthology.orchestra.run.smithi114.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-29T11:19:44.247 INFO:teuthology.orchestra.run.smithi114.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-29T11:19:44.247 INFO:teuthology.orchestra.run.smithi114.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-29T11:19:44.247 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T11:19:44.247 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-03-29T11:19:44.521 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph-fuse 2024-03-29T11:19:45.081 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:26 ago on Fri 29 Mar 2024 11:18:19 AM UTC. 2024-03-29T11:19:45.230 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-03-29T11:19:45.231 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-29T11:19:45.231 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repository Size 2024-03-29T11:19:45.231 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-29T11:19:45.231 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-03-29T11:19:45.231 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-29T11:19:45.231 INFO:teuthology.orchestra.run.smithi114.stdout:Installing dependencies: 2024-03-29T11:19:45.231 INFO:teuthology.orchestra.run.smithi114.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-29T11:19:45.231 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T11:19:45.231 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-03-29T11:19:45.231 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-29T11:19:45.231 INFO:teuthology.orchestra.run.smithi114.stdout:Install 2 Packages 2024-03-29T11:19:45.231 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T11:19:45.231 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 964 k 2024-03-29T11:19:45.232 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 2.8 M 2024-03-29T11:19:45.232 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-03-29T11:19:45.354 INFO:teuthology.orchestra.run.smithi114.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 646 kB/s | 79 kB 00:00 2024-03-29T11:19:45.570 INFO:teuthology.orchestra.run.smithi114.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 2.6 MB/s | 885 kB 00:00 2024-03-29T11:19:45.572 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:19:45.572 INFO:teuthology.orchestra.run.smithi114.stdout:Total 2.8 MB/s | 964 kB 00:00 2024-03-29T11:19:45.572 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-03-29T11:19:45.592 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-03-29T11:19:45.592 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-03-29T11:19:45.665 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-03-29T11:19:45.665 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-03-29T11:19:45.847 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-03-29T11:19:45.988 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-29T11:19:46.062 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:19:46.782 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:19:46.783 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-29T11:19:47.079 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-29T11:19:47.079 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T11:19:47.079 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-03-29T11:19:47.079 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-29T11:19:47.079 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T11:19:47.079 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-03-29T11:19:47.388 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install ceph-volume 2024-03-29T11:19:47.965 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:28 ago on Fri 29 Mar 2024 11:18:19 AM UTC. 2024-03-29T11:19:48.122 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-03-29T11:19:48.122 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-29T11:19:48.123 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repository Size 2024-03-29T11:19:48.123 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-29T11:19:48.123 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-03-29T11:19:48.123 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-29T11:19:48.123 INFO:teuthology.orchestra.run.smithi114.stdout:Installing dependencies: 2024-03-29T11:19:48.123 INFO:teuthology.orchestra.run.smithi114.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-29T11:19:48.123 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T11:19:48.123 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-03-29T11:19:48.123 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-29T11:19:48.123 INFO:teuthology.orchestra.run.smithi114.stdout:Install 2 Packages 2024-03-29T11:19:48.123 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T11:19:48.124 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 332 k 2024-03-29T11:19:48.124 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 1.5 M 2024-03-29T11:19:48.124 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-03-29T11:19:48.250 INFO:teuthology.orchestra.run.smithi114.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 544 kB/s | 69 kB 00:00 2024-03-29T11:19:48.400 INFO:teuthology.orchestra.run.smithi114.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 954 kB/s | 264 kB 00:00 2024-03-29T11:19:48.401 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:19:48.401 INFO:teuthology.orchestra.run.smithi114.stdout:Total 1.2 MB/s | 332 kB 00:00 2024-03-29T11:19:48.402 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-03-29T11:19:48.410 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-03-29T11:19:48.410 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-03-29T11:19:48.446 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-03-29T11:19:48.447 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-03-29T11:19:48.601 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-03-29T11:19:48.766 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-29T11:19:48.794 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:19:48.811 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:19:48.811 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:19:48.812 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-29T11:19:48.812 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T11:19:49.520 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-29T11:19:49.846 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-29T11:19:49.846 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T11:19:49.846 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-03-29T11:19:49.846 INFO:teuthology.orchestra.run.smithi114.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:19:49.847 INFO:teuthology.orchestra.run.smithi114.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-29T11:19:49.847 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T11:19:49.847 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-03-29T11:19:50.164 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install librados-devel 2024-03-29T11:19:50.739 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:31 ago on Fri 29 Mar 2024 11:18:19 AM UTC. 2024-03-29T11:19:50.890 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-03-29T11:19:50.891 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-29T11:19:50.891 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repo Size 2024-03-29T11:19:50.891 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-29T11:19:50.891 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-03-29T11:19:50.891 INFO:teuthology.orchestra.run.smithi114.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-29T11:19:50.891 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T11:19:50.891 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-03-29T11:19:50.891 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-29T11:19:50.891 INFO:teuthology.orchestra.run.smithi114.stdout:Install 1 Package 2024-03-29T11:19:50.891 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T11:19:50.891 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 127 k 2024-03-29T11:19:50.891 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 456 k 2024-03-29T11:19:50.892 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-03-29T11:19:51.197 INFO:teuthology.orchestra.run.smithi114.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 414 kB/s | 127 kB 00:00 2024-03-29T11:19:51.198 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:19:51.198 INFO:teuthology.orchestra.run.smithi114.stdout:Total 412 kB/s | 127 kB 00:00 2024-03-29T11:19:51.198 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-03-29T11:19:51.204 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-03-29T11:19:51.204 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-03-29T11:19:51.259 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-03-29T11:19:51.260 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-03-29T11:19:51.400 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-03-29T11:19:51.495 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:19:52.006 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:19:52.305 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:19:52.305 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T11:19:52.306 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-03-29T11:19:52.306 INFO:teuthology.orchestra.run.smithi114.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:19:52.306 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T11:19:52.306 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-03-29T11:19:52.566 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install libcephfs2 2024-03-29T11:19:53.144 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:34 ago on Fri 29 Mar 2024 11:18:19 AM UTC. 2024-03-29T11:19:53.247 INFO:teuthology.orchestra.run.smithi114.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:19:53.302 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-03-29T11:19:53.306 INFO:teuthology.orchestra.run.smithi114.stdout:Nothing to do. 2024-03-29T11:19:53.306 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-03-29T11:19:53.374 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install libcephfs-devel 2024-03-29T11:19:53.938 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:34 ago on Fri 29 Mar 2024 11:18:19 AM UTC. 2024-03-29T11:19:54.089 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-03-29T11:19:54.089 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-29T11:19:54.089 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repo Size 2024-03-29T11:19:54.090 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-29T11:19:54.090 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-03-29T11:19:54.090 INFO:teuthology.orchestra.run.smithi114.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-29T11:19:54.090 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T11:19:54.090 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-03-29T11:19:54.090 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-29T11:19:54.090 INFO:teuthology.orchestra.run.smithi114.stdout:Install 1 Package 2024-03-29T11:19:54.090 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T11:19:54.090 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 31 k 2024-03-29T11:19:54.090 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 140 k 2024-03-29T11:19:54.090 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-03-29T11:19:54.251 INFO:teuthology.orchestra.run.smithi114.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 194 kB/s | 31 kB 00:00 2024-03-29T11:19:54.251 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:19:54.251 INFO:teuthology.orchestra.run.smithi114.stdout:Total 193 kB/s | 31 kB 00:00 2024-03-29T11:19:54.251 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-03-29T11:19:54.255 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-03-29T11:19:54.255 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-03-29T11:19:54.296 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-03-29T11:19:54.297 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-03-29T11:19:54.428 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-03-29T11:19:54.515 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:19:54.957 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:19:55.218 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:19:55.218 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T11:19:55.218 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-03-29T11:19:55.218 INFO:teuthology.orchestra.run.smithi114.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:19:55.218 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T11:19:55.218 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-03-29T11:19:55.381 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install librados2 2024-03-29T11:19:55.946 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:36 ago on Fri 29 Mar 2024 11:18:19 AM UTC. 2024-03-29T11:19:56.042 INFO:teuthology.orchestra.run.smithi114.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:19:56.093 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-03-29T11:19:56.094 INFO:teuthology.orchestra.run.smithi114.stdout:Nothing to do. 2024-03-29T11:19:56.094 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-03-29T11:19:56.148 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install librbd1 2024-03-29T11:19:56.704 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:37 ago on Fri 29 Mar 2024 11:18:19 AM UTC. 2024-03-29T11:19:56.800 INFO:teuthology.orchestra.run.smithi114.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:19:56.851 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-03-29T11:19:56.852 INFO:teuthology.orchestra.run.smithi114.stdout:Nothing to do. 2024-03-29T11:19:56.852 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-03-29T11:19:56.902 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install python3-rados 2024-03-29T11:19:57.471 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:38 ago on Fri 29 Mar 2024 11:18:19 AM UTC. 2024-03-29T11:19:57.568 INFO:teuthology.orchestra.run.smithi114.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:19:57.621 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-03-29T11:19:57.622 INFO:teuthology.orchestra.run.smithi114.stdout:Nothing to do. 2024-03-29T11:19:57.622 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-03-29T11:19:57.666 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install python3-rgw 2024-03-29T11:19:58.229 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:39 ago on Fri 29 Mar 2024 11:18:19 AM UTC. 2024-03-29T11:19:58.327 INFO:teuthology.orchestra.run.smithi114.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:19:58.379 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-03-29T11:19:58.380 INFO:teuthology.orchestra.run.smithi114.stdout:Nothing to do. 2024-03-29T11:19:58.380 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-03-29T11:19:58.434 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install python3-cephfs 2024-03-29T11:19:59.001 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:40 ago on Fri 29 Mar 2024 11:18:19 AM UTC. 2024-03-29T11:19:59.099 INFO:teuthology.orchestra.run.smithi114.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:19:59.150 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-03-29T11:19:59.151 INFO:teuthology.orchestra.run.smithi114.stdout:Nothing to do. 2024-03-29T11:19:59.151 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-03-29T11:19:59.200 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install python3-rbd 2024-03-29T11:19:59.768 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:40 ago on Fri 29 Mar 2024 11:18:19 AM UTC. 2024-03-29T11:19:59.866 INFO:teuthology.orchestra.run.smithi114.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:19:59.919 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-03-29T11:19:59.920 INFO:teuthology.orchestra.run.smithi114.stdout:Nothing to do. 2024-03-29T11:19:59.920 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-03-29T11:19:59.981 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install rbd-fuse 2024-03-29T11:20:00.545 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:41 ago on Fri 29 Mar 2024 11:18:19 AM UTC. 2024-03-29T11:20:00.696 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-03-29T11:20:00.696 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-29T11:20:00.697 INFO:teuthology.orchestra.run.smithi114.stdout: Package Architecture Version Repository Size 2024-03-29T11:20:00.697 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-29T11:20:00.697 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-03-29T11:20:00.697 INFO:teuthology.orchestra.run.smithi114.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-29T11:20:00.697 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T11:20:00.697 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-03-29T11:20:00.697 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-29T11:20:00.697 INFO:teuthology.orchestra.run.smithi114.stdout:Install 1 Package 2024-03-29T11:20:00.697 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T11:20:00.697 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 85 k 2024-03-29T11:20:00.697 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 231 k 2024-03-29T11:20:00.697 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-03-29T11:20:01.012 INFO:teuthology.orchestra.run.smithi114.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 271 kB/s | 85 kB 00:00 2024-03-29T11:20:01.013 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:20:01.013 INFO:teuthology.orchestra.run.smithi114.stdout:Total 270 kB/s | 85 kB 00:00 2024-03-29T11:20:01.013 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-03-29T11:20:01.020 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-03-29T11:20:01.020 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-03-29T11:20:01.076 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-03-29T11:20:01.076 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-03-29T11:20:01.220 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-03-29T11:20:01.352 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:20:01.753 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:20:02.001 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:20:02.001 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T11:20:02.002 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-03-29T11:20:02.002 INFO:teuthology.orchestra.run.smithi114.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:20:02.002 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T11:20:02.002 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-03-29T11:20:02.262 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install rbd-mirror 2024-03-29T11:20:02.833 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:43 ago on Fri 29 Mar 2024 11:18:19 AM UTC. 2024-03-29T11:20:02.990 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-03-29T11:20:02.990 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-29T11:20:02.990 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repo Size 2024-03-29T11:20:02.991 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-29T11:20:02.991 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-03-29T11:20:02.991 INFO:teuthology.orchestra.run.smithi114.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:20:02.991 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T11:20:02.991 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-03-29T11:20:02.991 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-29T11:20:02.991 INFO:teuthology.orchestra.run.smithi114.stdout:Install 1 Package 2024-03-29T11:20:02.991 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T11:20:02.991 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 3.1 M 2024-03-29T11:20:02.991 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 13 M 2024-03-29T11:20:02.991 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-03-29T11:20:03.615 INFO:teuthology.orchestra.run.smithi114.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 5.0 MB/s | 3.1 MB 00:00 2024-03-29T11:20:03.615 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:20:03.616 INFO:teuthology.orchestra.run.smithi114.stdout:Total 4.9 MB/s | 3.1 MB 00:00 2024-03-29T11:20:03.616 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-03-29T11:20:03.625 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-03-29T11:20:03.625 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-03-29T11:20:03.699 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-03-29T11:20:03.699 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-03-29T11:20:04.130 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-03-29T11:20:04.191 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:20:04.215 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:20:04.215 INFO:teuthology.orchestra.run.smithi114.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:20:04.215 INFO:teuthology.orchestra.run.smithi114.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-29T11:20:04.216 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:20:04.216 INFO:teuthology.orchestra.run.smithi114.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:20:04.216 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T11:20:05.328 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:20:05.328 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T11:20:05.328 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-03-29T11:20:05.328 INFO:teuthology.orchestra.run.smithi114.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:20:05.328 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T11:20:05.328 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-03-29T11:20:05.569 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum -y install rbd-nbd 2024-03-29T11:20:06.145 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:01:47 ago on Fri 29 Mar 2024 11:18:19 AM UTC. 2024-03-29T11:20:06.294 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2024-03-29T11:20:06.295 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-29T11:20:06.295 INFO:teuthology.orchestra.run.smithi114.stdout: Package Architecture Version Repository Size 2024-03-29T11:20:06.295 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-29T11:20:06.295 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2024-03-29T11:20:06.296 INFO:teuthology.orchestra.run.smithi114.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-29T11:20:06.296 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T11:20:06.296 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2024-03-29T11:20:06.296 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2024-03-29T11:20:06.296 INFO:teuthology.orchestra.run.smithi114.stdout:Install 1 Package 2024-03-29T11:20:06.296 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T11:20:06.296 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 171 k 2024-03-29T11:20:06.296 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 494 k 2024-03-29T11:20:06.296 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2024-03-29T11:20:06.520 INFO:teuthology.orchestra.run.smithi114.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 766 kB/s | 171 kB 00:00 2024-03-29T11:20:06.521 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:20:06.521 INFO:teuthology.orchestra.run.smithi114.stdout:Total 761 kB/s | 171 kB 00:00 2024-03-29T11:20:06.521 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2024-03-29T11:20:06.529 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2024-03-29T11:20:06.529 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2024-03-29T11:20:06.586 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2024-03-29T11:20:06.586 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2024-03-29T11:20:06.734 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2024-03-29T11:20:06.836 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:20:07.305 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:20:07.595 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:20:07.596 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T11:20:07.596 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2024-03-29T11:20:07.596 INFO:teuthology.orchestra.run.smithi114.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:20:07.596 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T11:20:07.596 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2024-03-29T11:20:07.863 DEBUG:teuthology.parallel:result is None 2024-03-29T11:20:07.863 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:20:08.003 DEBUG:teuthology.orchestra.run.smithi114:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-29T11:20:08.024 INFO:teuthology.orchestra.run.smithi114.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-29T11:20:08.025 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-29T11:20:08.025 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-29T11:20:08.027 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:20:08.245 DEBUG:teuthology.orchestra.run.smithi176:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-29T11:20:08.270 INFO:teuthology.orchestra.run.smithi176.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-29T11:20:08.270 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-29T11:20:08.270 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-29T11:20:08.272 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-29T11:20:08.273 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-29T11:20:08.273 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-29T11:20:08.300 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-03-29T11:20:08.300 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-29T11:20:08.342 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-29T11:20:08.342 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-29T11:20:08.342 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/usr/bin/daemon-helper 2024-03-29T11:20:08.369 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-29T11:20:08.436 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-03-29T11:20:08.436 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/usr/bin/daemon-helper 2024-03-29T11:20:08.466 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-29T11:20:08.534 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-29T11:20:08.534 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-29T11:20:08.534 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-29T11:20:08.561 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-29T11:20:08.628 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-03-29T11:20:08.628 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-29T11:20:08.657 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-29T11:20:08.726 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-29T11:20:08.727 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-29T11:20:08.727 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/usr/bin/stdin-killer 2024-03-29T11:20:08.753 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-29T11:20:08.820 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-03-29T11:20:08.820 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/usr/bin/stdin-killer 2024-03-29T11:20:08.847 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-29T11:20:08.915 INFO:teuthology.run_tasks:Running task ceph... 2024-03-29T11:20:09.005 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-29T11:20:09.005 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 777 /var/log/ceph 2024-03-29T11:20:09.008 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod 777 /var/log/ceph 2024-03-29T11:20:09.036 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-29T11:20:09.036 DEBUG:teuthology.orchestra.run.smithi114:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-29T11:20:09.075 DEBUG:teuthology.orchestra.run.smithi176:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-29T11:20:09.108 INFO:tasks.ceph:Creating extra log directories... 2024-03-29T11:20:09.108 DEBUG:teuthology.orchestra.run.smithi114:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-29T11:20:09.141 DEBUG:teuthology.orchestra.run.smithi176:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-29T11:20:09.185 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-29T11:20:09.185 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-03-29T11:20:09.186 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622690', 'branch': 'main', 'description': 'rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados tasks/rados_api_tests validater/lockdep}', 'email': None, 'first_in_suite': False, 'job_id': '7622690', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request'], 'mon_bind_msgr2': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324'}}, 'owner': 'scheduled_amathuri@teuthology', 'priority': 150, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 8152, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'sleep_before_teardown': 0, 'subset': '99/1000', 'suite': 'rados:verify', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324', 'targets': {'smithi114.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBARmukIBI6GY5wbsB78CZNIyvRkpQev8UQ1peVGN117EtbV8WzHfJBDbP9V0zPW/fCCJVQ6y9x2ZLo7xriBIpQ8=', 'smithi176.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFy1Kl6VRcmodmVDtmzeaCCslSZmfSmeyp0er6tierAsPgfbqTAsjhhvSyWV3picBPdTnnn9YJHJGAOCuGtBLNU='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-26_05:45:44', 'tube': 'smithi', 'user': 'amathuri', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-29T11:20:09.186 DEBUG:teuthology.orchestra.run.smithi114:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-29T11:20:09.208 DEBUG:teuthology.orchestra.run.smithi176:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-29T11:20:09.231 DEBUG:teuthology.orchestra.run.smithi114:> sudo install -d -m0777 -- /var/run/ceph 2024-03-29T11:20:09.262 DEBUG:teuthology.orchestra.run.smithi176:> sudo install -d -m0777 -- /var/run/ceph 2024-03-29T11:20:09.299 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-29T11:20:09.300 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/scratch_devs of=/dev/stdout 2024-03-29T11:20:09.343 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-29T11:20:09.343 DEBUG:teuthology.orchestra.run.smithi114:> stat /dev/vg_nvme/lv_1 2024-03-29T11:20:09.397 INFO:teuthology.orchestra.run.smithi114.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-29T11:20:09.397 INFO:teuthology.orchestra.run.smithi114.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:20:09.397 INFO:teuthology.orchestra.run.smithi114.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-29T11:20:09.397 INFO:teuthology.orchestra.run.smithi114.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:20:09.397 INFO:teuthology.orchestra.run.smithi114.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:20:09.397 INFO:teuthology.orchestra.run.smithi114.stdout:Access: 2024-03-29 11:20:04.497730910 +0000 2024-03-29T11:20:09.397 INFO:teuthology.orchestra.run.smithi114.stdout:Modify: 2024-03-29 11:15:06.435091924 +0000 2024-03-29T11:20:09.397 INFO:teuthology.orchestra.run.smithi114.stdout:Change: 2024-03-29 11:15:06.435091924 +0000 2024-03-29T11:20:09.397 INFO:teuthology.orchestra.run.smithi114.stdout: Birth: 2024-03-29 11:15:06.435091924 +0000 2024-03-29T11:20:09.398 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-29T11:20:09.463 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records in 2024-03-29T11:20:09.463 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records out 2024-03-29T11:20:09.463 INFO:teuthology.orchestra.run.smithi114.stderr:512 bytes copied, 0.000234337 s, 2.2 MB/s 2024-03-29T11:20:09.464 DEBUG:teuthology.orchestra.run.smithi114:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-29T11:20:09.520 DEBUG:teuthology.orchestra.run.smithi114:> stat /dev/vg_nvme/lv_2 2024-03-29T11:20:09.575 INFO:teuthology.orchestra.run.smithi114.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-29T11:20:09.575 INFO:teuthology.orchestra.run.smithi114.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:20:09.576 INFO:teuthology.orchestra.run.smithi114.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-03-29T11:20:09.576 INFO:teuthology.orchestra.run.smithi114.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:20:09.576 INFO:teuthology.orchestra.run.smithi114.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:20:09.576 INFO:teuthology.orchestra.run.smithi114.stdout:Access: 2024-03-29 11:20:04.497730910 +0000 2024-03-29T11:20:09.576 INFO:teuthology.orchestra.run.smithi114.stdout:Modify: 2024-03-29 11:15:06.809084029 +0000 2024-03-29T11:20:09.576 INFO:teuthology.orchestra.run.smithi114.stdout:Change: 2024-03-29 11:15:06.809084029 +0000 2024-03-29T11:20:09.576 INFO:teuthology.orchestra.run.smithi114.stdout: Birth: 2024-03-29 11:15:06.809084029 +0000 2024-03-29T11:20:09.576 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-29T11:20:09.641 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records in 2024-03-29T11:20:09.641 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records out 2024-03-29T11:20:09.641 INFO:teuthology.orchestra.run.smithi114.stderr:512 bytes copied, 0.000207362 s, 2.5 MB/s 2024-03-29T11:20:09.642 DEBUG:teuthology.orchestra.run.smithi114:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-29T11:20:09.698 DEBUG:teuthology.orchestra.run.smithi114:> stat /dev/vg_nvme/lv_3 2024-03-29T11:20:09.753 INFO:teuthology.orchestra.run.smithi114.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-29T11:20:09.753 INFO:teuthology.orchestra.run.smithi114.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:20:09.753 INFO:teuthology.orchestra.run.smithi114.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-29T11:20:09.754 INFO:teuthology.orchestra.run.smithi114.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:20:09.754 INFO:teuthology.orchestra.run.smithi114.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:20:09.754 INFO:teuthology.orchestra.run.smithi114.stdout:Access: 2024-03-29 11:20:04.497730910 +0000 2024-03-29T11:20:09.754 INFO:teuthology.orchestra.run.smithi114.stdout:Modify: 2024-03-29 11:15:07.162076576 +0000 2024-03-29T11:20:09.754 INFO:teuthology.orchestra.run.smithi114.stdout:Change: 2024-03-29 11:15:07.162076576 +0000 2024-03-29T11:20:09.754 INFO:teuthology.orchestra.run.smithi114.stdout: Birth: 2024-03-29 11:15:07.162076576 +0000 2024-03-29T11:20:09.754 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-29T11:20:09.819 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records in 2024-03-29T11:20:09.819 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records out 2024-03-29T11:20:09.819 INFO:teuthology.orchestra.run.smithi114.stderr:512 bytes copied, 0.000186823 s, 2.7 MB/s 2024-03-29T11:20:09.821 DEBUG:teuthology.orchestra.run.smithi114:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-29T11:20:09.877 DEBUG:teuthology.orchestra.run.smithi114:> stat /dev/vg_nvme/lv_4 2024-03-29T11:20:09.931 INFO:teuthology.orchestra.run.smithi114.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-29T11:20:09.931 INFO:teuthology.orchestra.run.smithi114.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:20:09.931 INFO:teuthology.orchestra.run.smithi114.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-29T11:20:09.931 INFO:teuthology.orchestra.run.smithi114.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:20:09.931 INFO:teuthology.orchestra.run.smithi114.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:20:09.932 INFO:teuthology.orchestra.run.smithi114.stdout:Access: 2024-03-29 11:20:04.498730888 +0000 2024-03-29T11:20:09.932 INFO:teuthology.orchestra.run.smithi114.stdout:Modify: 2024-03-29 11:15:07.512069189 +0000 2024-03-29T11:20:09.932 INFO:teuthology.orchestra.run.smithi114.stdout:Change: 2024-03-29 11:15:07.512069189 +0000 2024-03-29T11:20:09.932 INFO:teuthology.orchestra.run.smithi114.stdout: Birth: 2024-03-29 11:15:07.512069189 +0000 2024-03-29T11:20:09.932 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-29T11:20:09.996 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records in 2024-03-29T11:20:09.997 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records out 2024-03-29T11:20:09.997 INFO:teuthology.orchestra.run.smithi114.stderr:512 bytes copied, 0.000237225 s, 2.2 MB/s 2024-03-29T11:20:09.998 DEBUG:teuthology.orchestra.run.smithi114:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-29T11:20:10.053 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:20:10.053 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-03-29T11:20:10.053 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/scratch_devs of=/dev/stdout 2024-03-29T11:20:10.069 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-29T11:20:10.069 DEBUG:teuthology.orchestra.run.smithi176:> stat /dev/vg_nvme/lv_1 2024-03-29T11:20:10.123 INFO:teuthology.orchestra.run.smithi176.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-29T11:20:10.123 INFO:teuthology.orchestra.run.smithi176.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:20:10.123 INFO:teuthology.orchestra.run.smithi176.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-03-29T11:20:10.124 INFO:teuthology.orchestra.run.smithi176.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:20:10.124 INFO:teuthology.orchestra.run.smithi176.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:20:10.124 INFO:teuthology.orchestra.run.smithi176.stdout:Access: 2024-03-29 11:18:25.575961349 +0000 2024-03-29T11:20:10.124 INFO:teuthology.orchestra.run.smithi176.stdout:Modify: 2024-03-29 11:14:56.072457919 +0000 2024-03-29T11:20:10.124 INFO:teuthology.orchestra.run.smithi176.stdout:Change: 2024-03-29 11:14:56.072457919 +0000 2024-03-29T11:20:10.124 INFO:teuthology.orchestra.run.smithi176.stdout: Birth: 2024-03-29 11:14:56.072457919 +0000 2024-03-29T11:20:10.124 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-29T11:20:10.189 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records in 2024-03-29T11:20:10.189 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records out 2024-03-29T11:20:10.189 INFO:teuthology.orchestra.run.smithi176.stderr:512 bytes copied, 0.000440621 s, 1.2 MB/s 2024-03-29T11:20:10.190 DEBUG:teuthology.orchestra.run.smithi176:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-29T11:20:10.246 DEBUG:teuthology.orchestra.run.smithi176:> stat /dev/vg_nvme/lv_2 2024-03-29T11:20:10.300 INFO:teuthology.orchestra.run.smithi176.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-29T11:20:10.300 INFO:teuthology.orchestra.run.smithi176.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:20:10.300 INFO:teuthology.orchestra.run.smithi176.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-03-29T11:20:10.300 INFO:teuthology.orchestra.run.smithi176.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:20:10.300 INFO:teuthology.orchestra.run.smithi176.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:20:10.300 INFO:teuthology.orchestra.run.smithi176.stdout:Access: 2024-03-29 11:18:25.575961349 +0000 2024-03-29T11:20:10.300 INFO:teuthology.orchestra.run.smithi176.stdout:Modify: 2024-03-29 11:14:56.403454837 +0000 2024-03-29T11:20:10.300 INFO:teuthology.orchestra.run.smithi176.stdout:Change: 2024-03-29 11:14:56.403454837 +0000 2024-03-29T11:20:10.300 INFO:teuthology.orchestra.run.smithi176.stdout: Birth: 2024-03-29 11:14:56.403454837 +0000 2024-03-29T11:20:10.301 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-29T11:20:10.366 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records in 2024-03-29T11:20:10.366 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records out 2024-03-29T11:20:10.366 INFO:teuthology.orchestra.run.smithi176.stderr:512 bytes copied, 0.000208962 s, 2.5 MB/s 2024-03-29T11:20:10.367 DEBUG:teuthology.orchestra.run.smithi176:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-29T11:20:10.423 DEBUG:teuthology.orchestra.run.smithi176:> stat /dev/vg_nvme/lv_3 2024-03-29T11:20:10.478 INFO:teuthology.orchestra.run.smithi176.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-29T11:20:10.478 INFO:teuthology.orchestra.run.smithi176.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:20:10.478 INFO:teuthology.orchestra.run.smithi176.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-03-29T11:20:10.479 INFO:teuthology.orchestra.run.smithi176.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:20:10.479 INFO:teuthology.orchestra.run.smithi176.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:20:10.479 INFO:teuthology.orchestra.run.smithi176.stdout:Access: 2024-03-29 11:18:25.576961339 +0000 2024-03-29T11:20:10.479 INFO:teuthology.orchestra.run.smithi176.stdout:Modify: 2024-03-29 11:14:56.763451485 +0000 2024-03-29T11:20:10.479 INFO:teuthology.orchestra.run.smithi176.stdout:Change: 2024-03-29 11:14:56.763451485 +0000 2024-03-29T11:20:10.479 INFO:teuthology.orchestra.run.smithi176.stdout: Birth: 2024-03-29 11:14:56.763451485 +0000 2024-03-29T11:20:10.479 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-29T11:20:10.543 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records in 2024-03-29T11:20:10.544 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records out 2024-03-29T11:20:10.544 INFO:teuthology.orchestra.run.smithi176.stderr:512 bytes copied, 0.00018572 s, 2.8 MB/s 2024-03-29T11:20:10.545 DEBUG:teuthology.orchestra.run.smithi176:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-29T11:20:10.601 DEBUG:teuthology.orchestra.run.smithi176:> stat /dev/vg_nvme/lv_4 2024-03-29T11:20:10.656 INFO:teuthology.orchestra.run.smithi176.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-29T11:20:10.656 INFO:teuthology.orchestra.run.smithi176.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:20:10.656 INFO:teuthology.orchestra.run.smithi176.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-29T11:20:10.656 INFO:teuthology.orchestra.run.smithi176.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:20:10.657 INFO:teuthology.orchestra.run.smithi176.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:20:10.657 INFO:teuthology.orchestra.run.smithi176.stdout:Access: 2024-03-29 11:18:25.576961339 +0000 2024-03-29T11:20:10.657 INFO:teuthology.orchestra.run.smithi176.stdout:Modify: 2024-03-29 11:14:57.107448281 +0000 2024-03-29T11:20:10.657 INFO:teuthology.orchestra.run.smithi176.stdout:Change: 2024-03-29 11:14:57.107448281 +0000 2024-03-29T11:20:10.657 INFO:teuthology.orchestra.run.smithi176.stdout: Birth: 2024-03-29 11:14:57.107448281 +0000 2024-03-29T11:20:10.657 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-29T11:20:10.722 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records in 2024-03-29T11:20:10.722 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records out 2024-03-29T11:20:10.722 INFO:teuthology.orchestra.run.smithi176.stderr:512 bytes copied, 0.000191764 s, 2.7 MB/s 2024-03-29T11:20:10.723 DEBUG:teuthology.orchestra.run.smithi176:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-29T11:20:10.779 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:20:10.779 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi114.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi176.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-29T11:20:10.779 INFO:tasks.ceph:Generating config... 2024-03-29T11:20:10.780 INFO:tasks.ceph:[client] debug monc = 20 2024-03-29T11:20:10.781 INFO:tasks.ceph:[client] debug ms = 1 2024-03-29T11:20:10.781 INFO:tasks.ceph:[client] debug objecter = 20 2024-03-29T11:20:10.781 INFO:tasks.ceph:[client] debug rados = 20 2024-03-29T11:20:10.781 INFO:tasks.ceph:[global] lockdep = True 2024-03-29T11:20:10.781 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-03-29T11:20:10.781 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-03-29T11:20:10.781 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-03-29T11:20:10.781 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-03-29T11:20:10.781 INFO:tasks.ceph:[global] ms type = async 2024-03-29T11:20:10.781 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-29T11:20:10.781 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-29T11:20:10.781 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-29T11:20:10.781 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-29T11:20:10.781 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-29T11:20:10.781 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-03-29T11:20:10.782 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-03-29T11:20:10.782 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-03-29T11:20:10.782 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-03-29T11:20:10.782 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-03-29T11:20:10.782 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-03-29T11:20:10.782 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-03-29T11:20:10.782 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-29T11:20:10.782 INFO:tasks.ceph:[osd] bluestore compression algorithm = snappy 2024-03-29T11:20:10.782 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-03-29T11:20:10.782 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-29T11:20:10.782 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-03-29T11:20:10.782 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-03-29T11:20:10.782 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-03-29T11:20:10.782 INFO:tasks.ceph:[osd] debug monc = 20 2024-03-29T11:20:10.783 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-29T11:20:10.783 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-29T11:20:10.783 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-29T11:20:10.783 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-29T11:20:10.783 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-29T11:20:10.783 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-29T11:20:10.783 INFO:tasks.ceph:[osd] osd class default list = * 2024-03-29T11:20:10.783 INFO:tasks.ceph:[osd] osd class load list = * 2024-03-29T11:20:10.783 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2024-03-29T11:20:10.783 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-03-29T11:20:10.783 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-03-29T11:20:10.784 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-29T11:20:10.784 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-03-29T11:20:10.784 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-03-29T11:20:10.784 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-29T11:20:10.784 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-03-29T11:20:10.784 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-03-29T11:20:10.784 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-03-29T11:20:10.784 INFO:tasks.ceph:Setting up mon.a... 2024-03-29T11:20:10.784 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-29T11:20:10.846 INFO:teuthology.orchestra.run.smithi114.stdout:creating /etc/ceph/ceph.keyring 2024-03-29T11:20:10.849 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-29T11:20:10.894 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T11:20:10.961 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.114:6789]'), ('mon.c', '[v1:172.21.15.114:6790]'), ('mon.b', '[v1:172.21.15.176:6789]')] 2024-03-29T11:20:10.961 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.114:6789],[v1:172.21.15.114:6790],[v1:172.21.15.176:6789]', 'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-03-29T11:20:10.962 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-29T11:20:10.962 DEBUG:teuthology.orchestra.run.smithi114:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-29T11:20:11.015 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.114:6789]' --addv c '[v1:172.21.15.114:6790]' --addv b '[v1:172.21.15.176:6789]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:20:11.099 INFO:teuthology.orchestra.run.smithi114.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:20:11.100 INFO:teuthology.orchestra.run.smithi114.stdout:monmaptool: generated fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:20:11.100 INFO:teuthology.orchestra.run.smithi114.stdout:setting min_mon_release = quincy 2024-03-29T11:20:11.100 INFO:teuthology.orchestra.run.smithi114.stdout:epoch 0 2024-03-29T11:20:11.100 INFO:teuthology.orchestra.run.smithi114.stdout:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:20:11.100 INFO:teuthology.orchestra.run.smithi114.stdout:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:11.101 INFO:teuthology.orchestra.run.smithi114.stdout:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:11.101 INFO:teuthology.orchestra.run.smithi114.stdout:min_mon_release 17 (quincy) 2024-03-29T11:20:11.101 INFO:teuthology.orchestra.run.smithi114.stdout:election_strategy: 1 2024-03-29T11:20:11.101 INFO:teuthology.orchestra.run.smithi114.stdout:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:20:11.101 INFO:teuthology.orchestra.run.smithi114.stdout:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:20:11.101 INFO:teuthology.orchestra.run.smithi114.stdout:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:20:11.101 INFO:teuthology.orchestra.run.smithi114.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-29T11:20:11.102 DEBUG:teuthology.orchestra.run.smithi114:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-29T11:20:11.157 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID b0de2b9b-c45a-44c7-b6ee-e83b39a1e108... 2024-03-29T11:20:11.158 DEBUG:teuthology.orchestra.run.smithi114:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-29T11:20:11.200 DEBUG:teuthology.orchestra.run.smithi176:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-29T11:20:11.247 INFO:teuthology.orchestra.run.smithi114.stdout:[global] 2024-03-29T11:20:11.247 INFO:teuthology.orchestra.run.smithi114.stdout: chdir = "" 2024-03-29T11:20:11.248 INFO:teuthology.orchestra.run.smithi114.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-29T11:20:11.248 INFO:teuthology.orchestra.run.smithi114.stdout: auth supported = cephx 2024-03-29T11:20:11.248 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T11:20:11.248 INFO:teuthology.orchestra.run.smithi114.stdout: filestore xattr use omap = true 2024-03-29T11:20:11.248 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T11:20:11.248 INFO:teuthology.orchestra.run.smithi114.stdout: mon clock drift allowed = 1.000 2024-03-29T11:20:11.248 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T11:20:11.248 INFO:teuthology.orchestra.run.smithi114.stdout: osd crush chooseleaf type = 0 2024-03-29T11:20:11.248 INFO:teuthology.orchestra.run.smithi114.stdout: auth debug = true 2024-03-29T11:20:11.248 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T11:20:11.248 INFO:teuthology.orchestra.run.smithi114.stdout: ms die on old message = true 2024-03-29T11:20:11.248 INFO:teuthology.orchestra.run.smithi114.stdout: ms die on bug = true 2024-03-29T11:20:11.248 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T11:20:11.248 INFO:teuthology.orchestra.run.smithi114.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-29T11:20:11.249 INFO:teuthology.orchestra.run.smithi114.stdout: mon pg warn max object skew = 0 2024-03-29T11:20:11.249 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T11:20:11.249 INFO:teuthology.orchestra.run.smithi114.stdout: # disable pg_autoscaler by default for new pools 2024-03-29T11:20:11.249 INFO:teuthology.orchestra.run.smithi114.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-29T11:20:11.249 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T11:20:11.249 INFO:teuthology.orchestra.run.smithi114.stdout: osd pool default size = 2 2024-03-29T11:20:11.249 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T11:20:11.249 INFO:teuthology.orchestra.run.smithi114.stdout: mon osd allow primary affinity = true 2024-03-29T11:20:11.249 INFO:teuthology.orchestra.run.smithi114.stdout: mon osd allow pg remap = true 2024-03-29T11:20:11.249 INFO:teuthology.orchestra.run.smithi114.stdout: mon warn on legacy crush tunables = false 2024-03-29T11:20:11.249 INFO:teuthology.orchestra.run.smithi114.stdout: mon warn on crush straw calc version zero = false 2024-03-29T11:20:11.249 INFO:teuthology.orchestra.run.smithi114.stdout: mon warn on no sortbitwise = false 2024-03-29T11:20:11.249 INFO:teuthology.orchestra.run.smithi114.stdout: mon warn on osd down out interval zero = false 2024-03-29T11:20:11.249 INFO:teuthology.orchestra.run.smithi114.stdout: mon warn on too few osds = false 2024-03-29T11:20:11.250 INFO:teuthology.orchestra.run.smithi114.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-29T11:20:11.250 INFO:teuthology.orchestra.run.smithi114.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-29T11:20:11.250 INFO:teuthology.orchestra.run.smithi114.stdout: mon_allow_pool_size_one = true 2024-03-29T11:20:11.250 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T11:20:11.250 INFO:teuthology.orchestra.run.smithi114.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-29T11:20:11.250 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T11:20:11.250 INFO:teuthology.orchestra.run.smithi114.stdout: osd default data pool replay window = 5 2024-03-29T11:20:11.250 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T11:20:11.250 INFO:teuthology.orchestra.run.smithi114.stdout: mon allow pool delete = true 2024-03-29T11:20:11.250 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T11:20:11.251 INFO:teuthology.orchestra.run.smithi114.stdout: mon cluster log file level = debug 2024-03-29T11:20:11.251 INFO:teuthology.orchestra.run.smithi114.stdout: debug asserts on shutdown = true 2024-03-29T11:20:11.251 INFO:teuthology.orchestra.run.smithi114.stdout: mon health detail to clog = false 2024-03-29T11:20:11.251 INFO:teuthology.orchestra.run.smithi114.stdout: mon host = "[v1:172.21.15.114:6789],[v1:172.21.15.114:6790],[v1:172.21.15.176:6789]" 2024-03-29T11:20:11.251 INFO:teuthology.orchestra.run.smithi114.stdout: lockdep = True 2024-03-29T11:20:11.251 INFO:teuthology.orchestra.run.smithi114.stdout: mon client directed command retry = 5 2024-03-29T11:20:11.251 INFO:teuthology.orchestra.run.smithi114.stdout: mon election default strategy = 1 2024-03-29T11:20:11.251 INFO:teuthology.orchestra.run.smithi114.stdout: ms bind msgr2 = False 2024-03-29T11:20:11.251 INFO:teuthology.orchestra.run.smithi114.stdout: ms inject socket failures = 5000 2024-03-29T11:20:11.251 INFO:teuthology.orchestra.run.smithi114.stdout: ms type = async 2024-03-29T11:20:11.251 INFO:teuthology.orchestra.run.smithi114.stdout: fsid = b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:20:11.251 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T11:20:11.251 INFO:teuthology.orchestra.run.smithi114.stdout:[osd] 2024-03-29T11:20:11.251 INFO:teuthology.orchestra.run.smithi114.stdout: osd journal size = 100 2024-03-29T11:20:11.251 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T11:20:11.252 INFO:teuthology.orchestra.run.smithi114.stdout: osd scrub load threshold = 5.0 2024-03-29T11:20:11.252 INFO:teuthology.orchestra.run.smithi114.stdout: osd scrub max interval = 600 2024-03-29T11:20:11.252 INFO:teuthology.orchestra.run.smithi114.stdout: osd mclock profile = high_recovery_ops 2024-03-29T11:20:11.252 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T11:20:11.252 INFO:teuthology.orchestra.run.smithi114.stdout: osd recover clone overlap = true 2024-03-29T11:20:11.252 INFO:teuthology.orchestra.run.smithi114.stdout: osd recovery max chunk = 1048576 2024-03-29T11:20:11.252 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T11:20:11.252 INFO:teuthology.orchestra.run.smithi114.stdout: osd debug shutdown = true 2024-03-29T11:20:11.252 INFO:teuthology.orchestra.run.smithi114.stdout: osd debug op order = true 2024-03-29T11:20:11.252 INFO:teuthology.orchestra.run.smithi114.stdout: osd debug verify stray on activate = true 2024-03-29T11:20:11.252 INFO:teuthology.orchestra.run.smithi114.stdout: osd debug trim objects = true 2024-03-29T11:20:11.252 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T11:20:11.252 INFO:teuthology.orchestra.run.smithi114.stdout: osd open classes on start = true 2024-03-29T11:20:11.252 INFO:teuthology.orchestra.run.smithi114.stdout: osd debug pg log writeout = true 2024-03-29T11:20:11.252 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T11:20:11.252 INFO:teuthology.orchestra.run.smithi114.stdout: osd deep scrub update digest min age = 30 2024-03-29T11:20:11.253 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T11:20:11.253 INFO:teuthology.orchestra.run.smithi114.stdout: osd map max advance = 10 2024-03-29T11:20:11.253 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T11:20:11.253 INFO:teuthology.orchestra.run.smithi114.stdout: journal zero on create = true 2024-03-29T11:20:11.253 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T11:20:11.253 INFO:teuthology.orchestra.run.smithi114.stdout: filestore ondisk finisher threads = 3 2024-03-29T11:20:11.253 INFO:teuthology.orchestra.run.smithi114.stdout: filestore apply finisher threads = 3 2024-03-29T11:20:11.253 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T11:20:11.253 INFO:teuthology.orchestra.run.smithi114.stdout: bdev debug aio = true 2024-03-29T11:20:11.253 INFO:teuthology.orchestra.run.smithi114.stdout: osd debug misdirected ops = true 2024-03-29T11:20:11.253 INFO:teuthology.orchestra.run.smithi114.stdout: bluestore block size = 96636764160 2024-03-29T11:20:11.253 INFO:teuthology.orchestra.run.smithi114.stdout: bluestore compression algorithm = snappy 2024-03-29T11:20:11.253 INFO:teuthology.orchestra.run.smithi114.stdout: bluestore compression mode = aggressive 2024-03-29T11:20:11.253 INFO:teuthology.orchestra.run.smithi114.stdout: bluestore fsck on mount = True 2024-03-29T11:20:11.254 INFO:teuthology.orchestra.run.smithi114.stdout: bluestore zero block detection = True 2024-03-29T11:20:11.254 INFO:teuthology.orchestra.run.smithi114.stdout: debug bluefs = 20 2024-03-29T11:20:11.254 INFO:teuthology.orchestra.run.smithi114.stdout: debug bluestore = 20 2024-03-29T11:20:11.254 INFO:teuthology.orchestra.run.smithi114.stdout: debug monc = 20 2024-03-29T11:20:11.254 INFO:teuthology.orchestra.run.smithi114.stdout: debug ms = 1 2024-03-29T11:20:11.254 INFO:teuthology.orchestra.run.smithi114.stdout: debug osd = 20 2024-03-29T11:20:11.254 INFO:teuthology.orchestra.run.smithi114.stdout: debug rocksdb = 10 2024-03-29T11:20:11.254 INFO:teuthology.orchestra.run.smithi114.stdout: mon osd backfillfull_ratio = 0.85 2024-03-29T11:20:11.254 INFO:teuthology.orchestra.run.smithi114.stdout: mon osd full ratio = 0.9 2024-03-29T11:20:11.254 INFO:teuthology.orchestra.run.smithi114.stdout: mon osd nearfull ratio = 0.8 2024-03-29T11:20:11.254 INFO:teuthology.orchestra.run.smithi114.stdout: osd class default list = * 2024-03-29T11:20:11.254 INFO:teuthology.orchestra.run.smithi114.stdout: osd class load list = * 2024-03-29T11:20:11.254 INFO:teuthology.orchestra.run.smithi114.stdout: osd client watch timeout = 120 2024-03-29T11:20:11.254 INFO:teuthology.orchestra.run.smithi114.stdout: osd debug verify cached snaps = True 2024-03-29T11:20:11.255 INFO:teuthology.orchestra.run.smithi114.stdout: osd debug verify missing on start = True 2024-03-29T11:20:11.255 INFO:teuthology.orchestra.run.smithi114.stdout: osd failsafe full ratio = 0.95 2024-03-29T11:20:11.255 INFO:teuthology.orchestra.run.smithi114.stdout: osd mclock override recovery settings = True 2024-03-29T11:20:11.255 INFO:teuthology.orchestra.run.smithi114.stdout: osd objectstore = bluestore 2024-03-29T11:20:11.255 INFO:teuthology.orchestra.run.smithi114.stdout: osd op queue = debug_random 2024-03-29T11:20:11.255 INFO:teuthology.orchestra.run.smithi114.stdout: osd op queue cut off = debug_random 2024-03-29T11:20:11.255 INFO:teuthology.orchestra.run.smithi114.stdout: osd shutdown pgref assert = True 2024-03-29T11:20:11.255 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T11:20:11.255 INFO:teuthology.orchestra.run.smithi114.stdout:[mgr] 2024-03-29T11:20:11.255 INFO:teuthology.orchestra.run.smithi114.stdout: debug ms = 1 2024-03-29T11:20:11.255 INFO:teuthology.orchestra.run.smithi114.stdout: debug mgr = 20 2024-03-29T11:20:11.255 INFO:teuthology.orchestra.run.smithi114.stdout: debug mon = 20 2024-03-29T11:20:11.255 INFO:teuthology.orchestra.run.smithi114.stdout: debug auth = 20 2024-03-29T11:20:11.256 INFO:teuthology.orchestra.run.smithi114.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:20:11.256 INFO:teuthology.orchestra.run.smithi114.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:20:11.256 INFO:teuthology.orchestra.run.smithi114.stdout: mgr/telemetry/nag = false 2024-03-29T11:20:11.256 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T11:20:11.256 INFO:teuthology.orchestra.run.smithi114.stdout:[mon] 2024-03-29T11:20:11.256 INFO:teuthology.orchestra.run.smithi114.stdout: debug ms = 1 2024-03-29T11:20:11.256 INFO:teuthology.orchestra.run.smithi114.stdout: debug mon = 20 2024-03-29T11:20:11.256 INFO:teuthology.orchestra.run.smithi114.stdout: debug paxos = 20 2024-03-29T11:20:11.256 INFO:teuthology.orchestra.run.smithi114.stdout: debug auth = 20 2024-03-29T11:20:11.256 INFO:teuthology.orchestra.run.smithi114.stdout: mon data avail warn = 5 2024-03-29T11:20:11.256 INFO:teuthology.orchestra.run.smithi114.stdout: mon mgr mkfs grace = 240 2024-03-29T11:20:11.256 INFO:teuthology.orchestra.run.smithi114.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:20:11.256 INFO:teuthology.orchestra.run.smithi114.stdout: mon osd reporter subtree level = osd 2024-03-29T11:20:11.256 INFO:teuthology.orchestra.run.smithi114.stdout: mon osd prime pg temp = true 2024-03-29T11:20:11.256 INFO:teuthology.orchestra.run.smithi114.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:20:11.257 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T11:20:11.257 INFO:teuthology.orchestra.run.smithi114.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-29T11:20:11.257 INFO:teuthology.orchestra.run.smithi114.stdout: auth mon ticket ttl = 660 # 11m 2024-03-29T11:20:11.257 INFO:teuthology.orchestra.run.smithi114.stdout: auth service ticket ttl = 240 # 4m 2024-03-29T11:20:11.257 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T11:20:11.257 INFO:teuthology.orchestra.run.smithi114.stdout: # don't complain about insecure global_id in the test suite 2024-03-29T11:20:11.257 INFO:teuthology.orchestra.run.smithi114.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-29T11:20:11.257 INFO:teuthology.orchestra.run.smithi114.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-29T11:20:11.257 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T11:20:11.257 INFO:teuthology.orchestra.run.smithi114.stdout: # 1m isn't quite enough 2024-03-29T11:20:11.257 INFO:teuthology.orchestra.run.smithi114.stdout: mon_down_mkfs_grace = 2m 2024-03-29T11:20:11.257 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T11:20:11.257 INFO:teuthology.orchestra.run.smithi114.stdout: mon_warn_on_filestore_osds = false 2024-03-29T11:20:11.257 INFO:teuthology.orchestra.run.smithi114.stdout: mon min osdmap epochs = 50 2024-03-29T11:20:11.257 INFO:teuthology.orchestra.run.smithi114.stdout: mon osdmap full prune interval = 2 2024-03-29T11:20:11.258 INFO:teuthology.orchestra.run.smithi114.stdout: mon osdmap full prune min = 15 2024-03-29T11:20:11.258 INFO:teuthology.orchestra.run.smithi114.stdout: mon osdmap full prune txsize = 2 2024-03-29T11:20:11.258 INFO:teuthology.orchestra.run.smithi114.stdout: mon scrub interval = 300 2024-03-29T11:20:11.258 INFO:teuthology.orchestra.run.smithi114.stdout: mon warn on pool no app = False 2024-03-29T11:20:11.258 INFO:teuthology.orchestra.run.smithi114.stdout: paxos service trim min = 10 2024-03-29T11:20:11.258 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T11:20:11.258 INFO:teuthology.orchestra.run.smithi114.stdout:[client] 2024-03-29T11:20:11.258 INFO:teuthology.orchestra.run.smithi114.stdout: rgw cache enabled = true 2024-03-29T11:20:11.258 INFO:teuthology.orchestra.run.smithi114.stdout: rgw enable ops log = true 2024-03-29T11:20:11.258 INFO:teuthology.orchestra.run.smithi114.stdout: rgw enable usage log = true 2024-03-29T11:20:11.258 INFO:teuthology.orchestra.run.smithi114.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-29T11:20:11.258 INFO:teuthology.orchestra.run.smithi114.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-29T11:20:11.258 INFO:teuthology.orchestra.run.smithi114.stdout: debug monc = 20 2024-03-29T11:20:11.258 INFO:teuthology.orchestra.run.smithi114.stdout: debug ms = 1 2024-03-29T11:20:11.258 INFO:teuthology.orchestra.run.smithi114.stdout: debug objecter = 20 2024-03-29T11:20:11.259 INFO:teuthology.orchestra.run.smithi114.stdout: debug rados = 20 2024-03-29T11:20:11.259 INFO:teuthology.orchestra.run.smithi114.stdout:[mon.a] 2024-03-29T11:20:11.259 INFO:teuthology.orchestra.run.smithi114.stdout:[mon.c] 2024-03-29T11:20:11.259 INFO:teuthology.orchestra.run.smithi114.stdout:[mon.b] 2024-03-29T11:20:11.259 INFO:teuthology.orchestra.run.smithi176.stdout:[global] 2024-03-29T11:20:11.259 INFO:teuthology.orchestra.run.smithi176.stdout: chdir = "" 2024-03-29T11:20:11.259 INFO:teuthology.orchestra.run.smithi176.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-29T11:20:11.259 INFO:teuthology.orchestra.run.smithi176.stdout: auth supported = cephx 2024-03-29T11:20:11.260 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T11:20:11.260 INFO:teuthology.orchestra.run.smithi176.stdout: filestore xattr use omap = true 2024-03-29T11:20:11.260 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T11:20:11.260 INFO:teuthology.orchestra.run.smithi176.stdout: mon clock drift allowed = 1.000 2024-03-29T11:20:11.260 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T11:20:11.260 INFO:teuthology.orchestra.run.smithi176.stdout: osd crush chooseleaf type = 0 2024-03-29T11:20:11.260 INFO:teuthology.orchestra.run.smithi176.stdout: auth debug = true 2024-03-29T11:20:11.260 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T11:20:11.260 INFO:teuthology.orchestra.run.smithi176.stdout: ms die on old message = true 2024-03-29T11:20:11.260 INFO:teuthology.orchestra.run.smithi176.stdout: ms die on bug = true 2024-03-29T11:20:11.260 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T11:20:11.260 INFO:teuthology.orchestra.run.smithi176.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-29T11:20:11.260 INFO:teuthology.orchestra.run.smithi176.stdout: mon pg warn max object skew = 0 2024-03-29T11:20:11.260 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T11:20:11.260 INFO:teuthology.orchestra.run.smithi176.stdout: # disable pg_autoscaler by default for new pools 2024-03-29T11:20:11.260 INFO:teuthology.orchestra.run.smithi176.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-29T11:20:11.261 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T11:20:11.261 INFO:teuthology.orchestra.run.smithi176.stdout: osd pool default size = 2 2024-03-29T11:20:11.261 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T11:20:11.261 INFO:teuthology.orchestra.run.smithi176.stdout: mon osd allow primary affinity = true 2024-03-29T11:20:11.261 INFO:teuthology.orchestra.run.smithi176.stdout: mon osd allow pg remap = true 2024-03-29T11:20:11.261 INFO:teuthology.orchestra.run.smithi176.stdout: mon warn on legacy crush tunables = false 2024-03-29T11:20:11.261 INFO:teuthology.orchestra.run.smithi176.stdout: mon warn on crush straw calc version zero = false 2024-03-29T11:20:11.261 INFO:teuthology.orchestra.run.smithi176.stdout: mon warn on no sortbitwise = false 2024-03-29T11:20:11.261 INFO:teuthology.orchestra.run.smithi176.stdout: mon warn on osd down out interval zero = false 2024-03-29T11:20:11.261 INFO:teuthology.orchestra.run.smithi176.stdout: mon warn on too few osds = false 2024-03-29T11:20:11.261 INFO:teuthology.orchestra.run.smithi176.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-29T11:20:11.261 INFO:teuthology.orchestra.run.smithi176.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-29T11:20:11.261 INFO:teuthology.orchestra.run.smithi176.stdout: mon_allow_pool_size_one = true 2024-03-29T11:20:11.261 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T11:20:11.261 INFO:teuthology.orchestra.run.smithi176.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-29T11:20:11.262 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T11:20:11.262 INFO:teuthology.orchestra.run.smithi176.stdout: osd default data pool replay window = 5 2024-03-29T11:20:11.262 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T11:20:11.262 INFO:teuthology.orchestra.run.smithi176.stdout: mon allow pool delete = true 2024-03-29T11:20:11.262 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T11:20:11.262 INFO:teuthology.orchestra.run.smithi176.stdout: mon cluster log file level = debug 2024-03-29T11:20:11.262 INFO:teuthology.orchestra.run.smithi176.stdout: debug asserts on shutdown = true 2024-03-29T11:20:11.262 INFO:teuthology.orchestra.run.smithi176.stdout: mon health detail to clog = false 2024-03-29T11:20:11.262 INFO:teuthology.orchestra.run.smithi176.stdout: mon host = "[v1:172.21.15.114:6789],[v1:172.21.15.114:6790],[v1:172.21.15.176:6789]" 2024-03-29T11:20:11.262 INFO:teuthology.orchestra.run.smithi176.stdout: lockdep = True 2024-03-29T11:20:11.262 INFO:teuthology.orchestra.run.smithi176.stdout: mon client directed command retry = 5 2024-03-29T11:20:11.262 INFO:teuthology.orchestra.run.smithi176.stdout: mon election default strategy = 1 2024-03-29T11:20:11.262 INFO:teuthology.orchestra.run.smithi176.stdout: ms bind msgr2 = False 2024-03-29T11:20:11.262 INFO:teuthology.orchestra.run.smithi176.stdout: ms inject socket failures = 5000 2024-03-29T11:20:11.262 INFO:teuthology.orchestra.run.smithi176.stdout: ms type = async 2024-03-29T11:20:11.263 INFO:teuthology.orchestra.run.smithi176.stdout: fsid = b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:20:11.263 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T11:20:11.263 INFO:teuthology.orchestra.run.smithi176.stdout:[osd] 2024-03-29T11:20:11.263 INFO:teuthology.orchestra.run.smithi176.stdout: osd journal size = 100 2024-03-29T11:20:11.263 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T11:20:11.263 INFO:teuthology.orchestra.run.smithi176.stdout: osd scrub load threshold = 5.0 2024-03-29T11:20:11.263 INFO:teuthology.orchestra.run.smithi176.stdout: osd scrub max interval = 600 2024-03-29T11:20:11.263 INFO:teuthology.orchestra.run.smithi176.stdout: osd mclock profile = high_recovery_ops 2024-03-29T11:20:11.263 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T11:20:11.263 INFO:teuthology.orchestra.run.smithi176.stdout: osd recover clone overlap = true 2024-03-29T11:20:11.263 INFO:teuthology.orchestra.run.smithi176.stdout: osd recovery max chunk = 1048576 2024-03-29T11:20:11.263 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T11:20:11.263 INFO:teuthology.orchestra.run.smithi176.stdout: osd debug shutdown = true 2024-03-29T11:20:11.263 INFO:teuthology.orchestra.run.smithi176.stdout: osd debug op order = true 2024-03-29T11:20:11.263 INFO:teuthology.orchestra.run.smithi176.stdout: osd debug verify stray on activate = true 2024-03-29T11:20:11.263 INFO:teuthology.orchestra.run.smithi176.stdout: osd debug trim objects = true 2024-03-29T11:20:11.264 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T11:20:11.264 INFO:teuthology.orchestra.run.smithi176.stdout: osd open classes on start = true 2024-03-29T11:20:11.264 INFO:teuthology.orchestra.run.smithi176.stdout: osd debug pg log writeout = true 2024-03-29T11:20:11.264 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T11:20:11.264 INFO:teuthology.orchestra.run.smithi176.stdout: osd deep scrub update digest min age = 30 2024-03-29T11:20:11.264 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T11:20:11.264 INFO:teuthology.orchestra.run.smithi176.stdout: osd map max advance = 10 2024-03-29T11:20:11.264 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T11:20:11.264 INFO:teuthology.orchestra.run.smithi176.stdout: journal zero on create = true 2024-03-29T11:20:11.264 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T11:20:11.264 INFO:teuthology.orchestra.run.smithi176.stdout: filestore ondisk finisher threads = 3 2024-03-29T11:20:11.264 INFO:teuthology.orchestra.run.smithi176.stdout: filestore apply finisher threads = 3 2024-03-29T11:20:11.264 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T11:20:11.264 INFO:teuthology.orchestra.run.smithi176.stdout: bdev debug aio = true 2024-03-29T11:20:11.264 INFO:teuthology.orchestra.run.smithi176.stdout: osd debug misdirected ops = true 2024-03-29T11:20:11.265 INFO:teuthology.orchestra.run.smithi176.stdout: bluestore block size = 96636764160 2024-03-29T11:20:11.265 INFO:teuthology.orchestra.run.smithi176.stdout: bluestore compression algorithm = snappy 2024-03-29T11:20:11.265 INFO:teuthology.orchestra.run.smithi176.stdout: bluestore compression mode = aggressive 2024-03-29T11:20:11.265 INFO:teuthology.orchestra.run.smithi176.stdout: bluestore fsck on mount = True 2024-03-29T11:20:11.265 INFO:teuthology.orchestra.run.smithi176.stdout: bluestore zero block detection = True 2024-03-29T11:20:11.265 INFO:teuthology.orchestra.run.smithi176.stdout: debug bluefs = 20 2024-03-29T11:20:11.265 INFO:teuthology.orchestra.run.smithi176.stdout: debug bluestore = 20 2024-03-29T11:20:11.265 INFO:teuthology.orchestra.run.smithi176.stdout: debug monc = 20 2024-03-29T11:20:11.265 INFO:teuthology.orchestra.run.smithi176.stdout: debug ms = 1 2024-03-29T11:20:11.265 INFO:teuthology.orchestra.run.smithi176.stdout: debug osd = 20 2024-03-29T11:20:11.265 INFO:teuthology.orchestra.run.smithi176.stdout: debug rocksdb = 10 2024-03-29T11:20:11.265 INFO:teuthology.orchestra.run.smithi176.stdout: mon osd backfillfull_ratio = 0.85 2024-03-29T11:20:11.265 INFO:teuthology.orchestra.run.smithi176.stdout: mon osd full ratio = 0.9 2024-03-29T11:20:11.265 INFO:teuthology.orchestra.run.smithi176.stdout: mon osd nearfull ratio = 0.8 2024-03-29T11:20:11.265 INFO:teuthology.orchestra.run.smithi176.stdout: osd class default list = * 2024-03-29T11:20:11.265 INFO:teuthology.orchestra.run.smithi176.stdout: osd class load list = * 2024-03-29T11:20:11.266 INFO:teuthology.orchestra.run.smithi176.stdout: osd client watch timeout = 120 2024-03-29T11:20:11.266 INFO:teuthology.orchestra.run.smithi176.stdout: osd debug verify cached snaps = True 2024-03-29T11:20:11.266 INFO:teuthology.orchestra.run.smithi176.stdout: osd debug verify missing on start = True 2024-03-29T11:20:11.266 INFO:teuthology.orchestra.run.smithi176.stdout: osd failsafe full ratio = 0.95 2024-03-29T11:20:11.266 INFO:teuthology.orchestra.run.smithi176.stdout: osd mclock override recovery settings = True 2024-03-29T11:20:11.266 INFO:teuthology.orchestra.run.smithi176.stdout: osd objectstore = bluestore 2024-03-29T11:20:11.266 INFO:teuthology.orchestra.run.smithi176.stdout: osd op queue = debug_random 2024-03-29T11:20:11.266 INFO:teuthology.orchestra.run.smithi176.stdout: osd op queue cut off = debug_random 2024-03-29T11:20:11.266 INFO:teuthology.orchestra.run.smithi176.stdout: osd shutdown pgref assert = True 2024-03-29T11:20:11.266 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T11:20:11.266 INFO:teuthology.orchestra.run.smithi176.stdout:[mgr] 2024-03-29T11:20:11.266 INFO:teuthology.orchestra.run.smithi176.stdout: debug ms = 1 2024-03-29T11:20:11.266 INFO:teuthology.orchestra.run.smithi176.stdout: debug mgr = 20 2024-03-29T11:20:11.267 INFO:teuthology.orchestra.run.smithi176.stdout: debug mon = 20 2024-03-29T11:20:11.267 INFO:teuthology.orchestra.run.smithi176.stdout: debug auth = 20 2024-03-29T11:20:11.267 INFO:teuthology.orchestra.run.smithi176.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:20:11.267 INFO:teuthology.orchestra.run.smithi176.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:20:11.267 INFO:teuthology.orchestra.run.smithi176.stdout: mgr/telemetry/nag = false 2024-03-29T11:20:11.267 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T11:20:11.267 INFO:teuthology.orchestra.run.smithi176.stdout:[mon] 2024-03-29T11:20:11.267 INFO:teuthology.orchestra.run.smithi176.stdout: debug ms = 1 2024-03-29T11:20:11.267 INFO:teuthology.orchestra.run.smithi176.stdout: debug mon = 20 2024-03-29T11:20:11.267 INFO:teuthology.orchestra.run.smithi176.stdout: debug paxos = 20 2024-03-29T11:20:11.267 INFO:teuthology.orchestra.run.smithi176.stdout: debug auth = 20 2024-03-29T11:20:11.267 INFO:teuthology.orchestra.run.smithi176.stdout: mon data avail warn = 5 2024-03-29T11:20:11.267 INFO:teuthology.orchestra.run.smithi176.stdout: mon mgr mkfs grace = 240 2024-03-29T11:20:11.268 INFO:teuthology.orchestra.run.smithi176.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:20:11.268 INFO:teuthology.orchestra.run.smithi176.stdout: mon osd reporter subtree level = osd 2024-03-29T11:20:11.268 INFO:teuthology.orchestra.run.smithi176.stdout: mon osd prime pg temp = true 2024-03-29T11:20:11.268 INFO:teuthology.orchestra.run.smithi176.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:20:11.268 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T11:20:11.268 INFO:teuthology.orchestra.run.smithi176.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-29T11:20:11.268 INFO:teuthology.orchestra.run.smithi176.stdout: auth mon ticket ttl = 660 # 11m 2024-03-29T11:20:11.269 INFO:teuthology.orchestra.run.smithi176.stdout: auth service ticket ttl = 240 # 4m 2024-03-29T11:20:11.269 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T11:20:11.269 INFO:teuthology.orchestra.run.smithi176.stdout: # don't complain about insecure global_id in the test suite 2024-03-29T11:20:11.269 INFO:teuthology.orchestra.run.smithi176.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-29T11:20:11.269 INFO:teuthology.orchestra.run.smithi176.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-29T11:20:11.269 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T11:20:11.269 INFO:teuthology.orchestra.run.smithi176.stdout: # 1m isn't quite enough 2024-03-29T11:20:11.270 INFO:teuthology.orchestra.run.smithi176.stdout: mon_down_mkfs_grace = 2m 2024-03-29T11:20:11.270 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T11:20:11.270 INFO:teuthology.orchestra.run.smithi176.stdout: mon_warn_on_filestore_osds = false 2024-03-29T11:20:11.270 INFO:teuthology.orchestra.run.smithi176.stdout: mon min osdmap epochs = 50 2024-03-29T11:20:11.270 INFO:teuthology.orchestra.run.smithi176.stdout: mon osdmap full prune interval = 2 2024-03-29T11:20:11.270 INFO:teuthology.orchestra.run.smithi176.stdout: mon osdmap full prune min = 15 2024-03-29T11:20:11.271 INFO:teuthology.orchestra.run.smithi176.stdout: mon osdmap full prune txsize = 2 2024-03-29T11:20:11.271 INFO:teuthology.orchestra.run.smithi176.stdout: mon scrub interval = 300 2024-03-29T11:20:11.271 INFO:teuthology.orchestra.run.smithi176.stdout: mon warn on pool no app = False 2024-03-29T11:20:11.271 INFO:teuthology.orchestra.run.smithi176.stdout: paxos service trim min = 10 2024-03-29T11:20:11.271 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-29T11:20:11.271 INFO:teuthology.orchestra.run.smithi176.stdout:[client] 2024-03-29T11:20:11.271 INFO:teuthology.orchestra.run.smithi176.stdout: rgw cache enabled = true 2024-03-29T11:20:11.272 INFO:teuthology.orchestra.run.smithi176.stdout: rgw enable ops log = true 2024-03-29T11:20:11.272 INFO:teuthology.orchestra.run.smithi176.stdout: rgw enable usage log = true 2024-03-29T11:20:11.272 INFO:teuthology.orchestra.run.smithi176.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-29T11:20:11.272 INFO:teuthology.orchestra.run.smithi176.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-29T11:20:11.272 INFO:teuthology.orchestra.run.smithi176.stdout: debug monc = 20 2024-03-29T11:20:11.272 INFO:teuthology.orchestra.run.smithi176.stdout: debug ms = 1 2024-03-29T11:20:11.272 INFO:teuthology.orchestra.run.smithi176.stdout: debug objecter = 20 2024-03-29T11:20:11.273 INFO:teuthology.orchestra.run.smithi176.stdout: debug rados = 20 2024-03-29T11:20:11.273 INFO:teuthology.orchestra.run.smithi176.stdout:[mon.a] 2024-03-29T11:20:11.273 INFO:teuthology.orchestra.run.smithi176.stdout:[mon.c] 2024-03-29T11:20:11.273 INFO:teuthology.orchestra.run.smithi176.stdout:[mon.b] 2024-03-29T11:20:11.275 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-29T11:20:11.276 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-29T11:20:11.370 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-29T11:20:11.371 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-29T11:20:11.371 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-29T11:20:11.385 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-29T11:20:11.385 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-29T11:20:11.440 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi114.front.sepia.ceph.com 2024-03-29T11:20:11.440 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-29T11:20:11.441 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-29T11:20:11.441 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T11:20:11.519 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-29T11:20:11.520 DEBUG:teuthology.orchestra.run.smithi114:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:20:11.574 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi176.front.sepia.ceph.com 2024-03-29T11:20:11.574 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-03-29T11:20:11.575 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-29T11:20:11.575 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T11:20:11.614 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-03-29T11:20:11.614 DEBUG:teuthology.orchestra.run.smithi176:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:20:11.668 INFO:tasks.ceph:Setting up mon nodes... 2024-03-29T11:20:11.668 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-29T11:20:11.668 DEBUG:teuthology.orchestra.run.smithi114:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-29T11:20:11.723 INFO:teuthology.orchestra.run.smithi114.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-29T11:20:11.726 DEBUG:teuthology.orchestra.run.smithi176:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-29T11:20:11.791 INFO:teuthology.orchestra.run.smithi176.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-29T11:20:11.794 INFO:tasks.ceph:Setting up mds nodes... 2024-03-29T11:20:11.794 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-29T11:20:11.794 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-29T11:20:11.838 INFO:teuthology.orchestra.run.smithi114.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-29T11:20:11.853 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-29T11:20:11.895 INFO:teuthology.orchestra.run.smithi176.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-29T11:20:11.910 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-29T11:20:11.910 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi114.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi176.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-29T11:20:11.910 DEBUG:teuthology.orchestra.run.smithi114:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-29T11:20:11.938 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:20:11.938 INFO:tasks.ceph:role: osd.0 2024-03-29T11:20:11.939 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi114.front.sepia.ceph.com 2024-03-29T11:20:11.939 DEBUG:teuthology.orchestra.run.smithi114:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-29T11:20:12.012 INFO:teuthology.orchestra.run.smithi114.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:20:12.012 INFO:teuthology.orchestra.run.smithi114.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:20:12.012 INFO:teuthology.orchestra.run.smithi114.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:20:12.012 INFO:teuthology.orchestra.run.smithi114.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:20:12.013 INFO:teuthology.orchestra.run.smithi114.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:20:12.013 INFO:teuthology.orchestra.run.smithi114.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:20:12.013 INFO:teuthology.orchestra.run.smithi114.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:20:12.013 INFO:teuthology.orchestra.run.smithi114.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:20:12.013 INFO:teuthology.orchestra.run.smithi114.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:20:12.013 INFO:teuthology.orchestra.run.smithi114.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:20:12.516 INFO:teuthology.orchestra.run.smithi114.stdout:Discarding blocks...Done. 2024-03-29T11:20:12.518 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi114.front.sepia.ceph.com -o noatime 2024-03-29T11:20:12.518 DEBUG:teuthology.orchestra.run.smithi114:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-29T11:20:12.648 DEBUG:teuthology.orchestra.run.smithi114:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-29T11:20:12.682 DEBUG:teuthology.orchestra.run.smithi114:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-29T11:20:12.750 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:20:12.750 INFO:tasks.ceph:role: osd.1 2024-03-29T11:20:12.750 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi114.front.sepia.ceph.com 2024-03-29T11:20:12.751 DEBUG:teuthology.orchestra.run.smithi114:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-29T11:20:12.825 INFO:teuthology.orchestra.run.smithi114.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:20:12.825 INFO:teuthology.orchestra.run.smithi114.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:20:12.826 INFO:teuthology.orchestra.run.smithi114.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:20:12.826 INFO:teuthology.orchestra.run.smithi114.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:20:12.826 INFO:teuthology.orchestra.run.smithi114.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:20:12.826 INFO:teuthology.orchestra.run.smithi114.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:20:12.826 INFO:teuthology.orchestra.run.smithi114.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:20:12.826 INFO:teuthology.orchestra.run.smithi114.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:20:12.826 INFO:teuthology.orchestra.run.smithi114.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:20:12.826 INFO:teuthology.orchestra.run.smithi114.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:20:13.303 INFO:teuthology.orchestra.run.smithi114.stdout:Discarding blocks...Done. 2024-03-29T11:20:13.305 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi114.front.sepia.ceph.com -o noatime 2024-03-29T11:20:13.306 DEBUG:teuthology.orchestra.run.smithi114:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-29T11:20:13.401 DEBUG:teuthology.orchestra.run.smithi114:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-29T11:20:13.433 DEBUG:teuthology.orchestra.run.smithi114:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-29T11:20:13.501 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:20:13.501 INFO:tasks.ceph:role: osd.2 2024-03-29T11:20:13.501 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi114.front.sepia.ceph.com 2024-03-29T11:20:13.501 DEBUG:teuthology.orchestra.run.smithi114:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-29T11:20:13.578 INFO:teuthology.orchestra.run.smithi114.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:20:13.578 INFO:teuthology.orchestra.run.smithi114.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:20:13.578 INFO:teuthology.orchestra.run.smithi114.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:20:13.578 INFO:teuthology.orchestra.run.smithi114.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:20:13.578 INFO:teuthology.orchestra.run.smithi114.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:20:13.578 INFO:teuthology.orchestra.run.smithi114.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:20:13.578 INFO:teuthology.orchestra.run.smithi114.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:20:13.579 INFO:teuthology.orchestra.run.smithi114.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:20:13.579 INFO:teuthology.orchestra.run.smithi114.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:20:13.579 INFO:teuthology.orchestra.run.smithi114.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:20:14.072 INFO:teuthology.orchestra.run.smithi114.stdout:Discarding blocks...Done. 2024-03-29T11:20:14.074 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi114.front.sepia.ceph.com -o noatime 2024-03-29T11:20:14.074 DEBUG:teuthology.orchestra.run.smithi114:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-29T11:20:14.237 DEBUG:teuthology.orchestra.run.smithi114:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-29T11:20:14.271 DEBUG:teuthology.orchestra.run.smithi114:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-29T11:20:14.340 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:20:14.340 INFO:tasks.ceph:role: osd.3 2024-03-29T11:20:14.340 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi114.front.sepia.ceph.com 2024-03-29T11:20:14.340 DEBUG:teuthology.orchestra.run.smithi114:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-29T11:20:14.415 INFO:teuthology.orchestra.run.smithi114.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:20:14.415 INFO:teuthology.orchestra.run.smithi114.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:20:14.415 INFO:teuthology.orchestra.run.smithi114.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:20:14.415 INFO:teuthology.orchestra.run.smithi114.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:20:14.415 INFO:teuthology.orchestra.run.smithi114.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:20:14.415 INFO:teuthology.orchestra.run.smithi114.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:20:14.416 INFO:teuthology.orchestra.run.smithi114.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:20:14.416 INFO:teuthology.orchestra.run.smithi114.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:20:14.416 INFO:teuthology.orchestra.run.smithi114.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:20:14.416 INFO:teuthology.orchestra.run.smithi114.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:20:14.927 INFO:teuthology.orchestra.run.smithi114.stdout:Discarding blocks...Done. 2024-03-29T11:20:14.929 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi114.front.sepia.ceph.com -o noatime 2024-03-29T11:20:14.929 DEBUG:teuthology.orchestra.run.smithi114:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-29T11:20:15.038 DEBUG:teuthology.orchestra.run.smithi114:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-29T11:20:15.074 DEBUG:teuthology.orchestra.run.smithi114:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:20:15.173 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:15.196+0000 7fd5aec17640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-29T11:20:15.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:15.196+0000 7fd5aec17640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-29T11:20:15.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:15.196+0000 7fd5aec17640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T11:20:15.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:15.196+0000 7fd5aec17640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T11:20:15.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:15.196+0000 7fd5aec17640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T11:20:15.174 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:15.197+0000 7fd5aec17640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-29T11:20:18.029 DEBUG:teuthology.orchestra.run.smithi114:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-29T11:20:18.057 DEBUG:teuthology.orchestra.run.smithi114:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:20:18.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:18.167+0000 7f1715e17640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-29T11:20:18.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:18.167+0000 7f1715e17640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-29T11:20:18.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:18.167+0000 7f1715e17640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T11:20:18.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:18.167+0000 7f1715e17640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T11:20:18.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:18.167+0000 7f1715e17640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T11:20:18.144 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:18.167+0000 7f1715e17640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-29T11:20:21.002 DEBUG:teuthology.orchestra.run.smithi114:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-29T11:20:21.032 DEBUG:teuthology.orchestra.run.smithi114:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:20:21.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:21.145+0000 7ff05a80f640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-29T11:20:21.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:21.146+0000 7ff05a80f640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-29T11:20:21.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:21.146+0000 7ff05a80f640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T11:20:21.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:21.146+0000 7ff05a80f640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T11:20:21.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:21.146+0000 7ff05a80f640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T11:20:21.123 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:21.146+0000 7ff05a80f640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-29T11:20:23.992 DEBUG:teuthology.orchestra.run.smithi114:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-29T11:20:24.022 DEBUG:teuthology.orchestra.run.smithi114:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:20:24.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:24.136+0000 7fa8edbf9640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-29T11:20:24.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:24.136+0000 7fa8edbf9640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-29T11:20:24.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:24.136+0000 7fa8edbf9640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T11:20:24.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:24.136+0000 7fa8edbf9640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T11:20:24.113 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:24.136+0000 7fa8edbf9640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T11:20:24.114 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:24.137+0000 7fa8edbf9640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-29T11:20:26.975 DEBUG:teuthology.orchestra.run.smithi114:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-29T11:20:27.006 DEBUG:teuthology.orchestra.run.smithi176:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-29T11:20:27.038 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:20:27.038 INFO:tasks.ceph:role: osd.4 2024-03-29T11:20:27.038 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi176.front.sepia.ceph.com 2024-03-29T11:20:27.038 DEBUG:teuthology.orchestra.run.smithi176:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-29T11:20:27.116 INFO:teuthology.orchestra.run.smithi176.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:20:27.116 INFO:teuthology.orchestra.run.smithi176.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:20:27.116 INFO:teuthology.orchestra.run.smithi176.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:20:27.116 INFO:teuthology.orchestra.run.smithi176.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:20:27.116 INFO:teuthology.orchestra.run.smithi176.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:20:27.116 INFO:teuthology.orchestra.run.smithi176.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:20:27.116 INFO:teuthology.orchestra.run.smithi176.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:20:27.116 INFO:teuthology.orchestra.run.smithi176.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:20:27.116 INFO:teuthology.orchestra.run.smithi176.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:20:27.117 INFO:teuthology.orchestra.run.smithi176.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:20:27.635 INFO:teuthology.orchestra.run.smithi176.stdout:Discarding blocks...Done. 2024-03-29T11:20:27.637 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi176.front.sepia.ceph.com -o noatime 2024-03-29T11:20:27.638 DEBUG:teuthology.orchestra.run.smithi176:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-29T11:20:27.768 DEBUG:teuthology.orchestra.run.smithi176:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-29T11:20:27.805 DEBUG:teuthology.orchestra.run.smithi176:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-29T11:20:27.874 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:20:27.875 INFO:tasks.ceph:role: osd.5 2024-03-29T11:20:27.875 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi176.front.sepia.ceph.com 2024-03-29T11:20:27.875 DEBUG:teuthology.orchestra.run.smithi176:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-29T11:20:27.950 INFO:teuthology.orchestra.run.smithi176.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:20:27.950 INFO:teuthology.orchestra.run.smithi176.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:20:27.950 INFO:teuthology.orchestra.run.smithi176.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:20:27.950 INFO:teuthology.orchestra.run.smithi176.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:20:27.950 INFO:teuthology.orchestra.run.smithi176.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:20:27.951 INFO:teuthology.orchestra.run.smithi176.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:20:27.951 INFO:teuthology.orchestra.run.smithi176.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:20:27.951 INFO:teuthology.orchestra.run.smithi176.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:20:27.951 INFO:teuthology.orchestra.run.smithi176.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:20:27.951 INFO:teuthology.orchestra.run.smithi176.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:20:28.399 INFO:teuthology.orchestra.run.smithi176.stdout:Discarding blocks...Done. 2024-03-29T11:20:28.401 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi176.front.sepia.ceph.com -o noatime 2024-03-29T11:20:28.401 DEBUG:teuthology.orchestra.run.smithi176:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-29T11:20:28.497 DEBUG:teuthology.orchestra.run.smithi176:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-29T11:20:28.530 DEBUG:teuthology.orchestra.run.smithi176:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-29T11:20:28.599 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:20:28.599 INFO:tasks.ceph:role: osd.6 2024-03-29T11:20:28.599 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi176.front.sepia.ceph.com 2024-03-29T11:20:28.599 DEBUG:teuthology.orchestra.run.smithi176:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-29T11:20:28.674 INFO:teuthology.orchestra.run.smithi176.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:20:28.675 INFO:teuthology.orchestra.run.smithi176.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:20:28.675 INFO:teuthology.orchestra.run.smithi176.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:20:28.675 INFO:teuthology.orchestra.run.smithi176.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:20:28.675 INFO:teuthology.orchestra.run.smithi176.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:20:28.675 INFO:teuthology.orchestra.run.smithi176.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:20:28.675 INFO:teuthology.orchestra.run.smithi176.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:20:28.675 INFO:teuthology.orchestra.run.smithi176.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:20:28.675 INFO:teuthology.orchestra.run.smithi176.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:20:28.675 INFO:teuthology.orchestra.run.smithi176.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:20:29.156 INFO:teuthology.orchestra.run.smithi176.stdout:Discarding blocks...Done. 2024-03-29T11:20:29.159 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi176.front.sepia.ceph.com -o noatime 2024-03-29T11:20:29.159 DEBUG:teuthology.orchestra.run.smithi176:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-29T11:20:29.313 DEBUG:teuthology.orchestra.run.smithi176:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-29T11:20:29.347 DEBUG:teuthology.orchestra.run.smithi176:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-29T11:20:29.416 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:20:29.416 INFO:tasks.ceph:role: osd.7 2024-03-29T11:20:29.416 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi176.front.sepia.ceph.com 2024-03-29T11:20:29.416 DEBUG:teuthology.orchestra.run.smithi176:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-29T11:20:29.492 INFO:teuthology.orchestra.run.smithi176.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:20:29.492 INFO:teuthology.orchestra.run.smithi176.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:20:29.492 INFO:teuthology.orchestra.run.smithi176.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:20:29.492 INFO:teuthology.orchestra.run.smithi176.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:20:29.492 INFO:teuthology.orchestra.run.smithi176.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:20:29.493 INFO:teuthology.orchestra.run.smithi176.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:20:29.493 INFO:teuthology.orchestra.run.smithi176.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:20:29.493 INFO:teuthology.orchestra.run.smithi176.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:20:29.493 INFO:teuthology.orchestra.run.smithi176.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:20:29.493 INFO:teuthology.orchestra.run.smithi176.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:20:30.007 INFO:teuthology.orchestra.run.smithi176.stdout:Discarding blocks...Done. 2024-03-29T11:20:30.009 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi176.front.sepia.ceph.com -o noatime 2024-03-29T11:20:30.009 DEBUG:teuthology.orchestra.run.smithi176:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-29T11:20:30.153 DEBUG:teuthology.orchestra.run.smithi176:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-29T11:20:30.189 DEBUG:teuthology.orchestra.run.smithi176:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:20:30.295 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:30.318+0000 7f6e52eb5640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-29T11:20:30.295 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:30.318+0000 7f6e52eb5640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-29T11:20:30.296 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:30.318+0000 7f6e52eb5640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T11:20:30.296 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:30.318+0000 7f6e52eb5640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T11:20:30.296 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:30.318+0000 7f6e52eb5640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T11:20:30.296 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:30.319+0000 7f6e52eb5640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-29T11:20:33.153 DEBUG:teuthology.orchestra.run.smithi176:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-29T11:20:33.185 DEBUG:teuthology.orchestra.run.smithi176:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:20:33.275 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:33.298+0000 7f0ad7549640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-29T11:20:33.275 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:33.298+0000 7f0ad7549640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-29T11:20:33.275 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:33.298+0000 7f0ad7549640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T11:20:33.275 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:33.298+0000 7f0ad7549640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T11:20:33.276 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:33.298+0000 7f0ad7549640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T11:20:33.276 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:33.299+0000 7f0ad7549640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-29T11:20:36.125 DEBUG:teuthology.orchestra.run.smithi176:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-29T11:20:36.156 DEBUG:teuthology.orchestra.run.smithi176:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:20:36.248 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:36.271+0000 7fbaf1947640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-29T11:20:36.248 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:36.271+0000 7fbaf1947640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-29T11:20:36.249 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:36.271+0000 7fbaf1947640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T11:20:36.249 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:36.271+0000 7fbaf1947640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T11:20:36.249 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:36.271+0000 7fbaf1947640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T11:20:36.249 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:36.272+0000 7fbaf1947640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-29T11:20:39.105 DEBUG:teuthology.orchestra.run.smithi176:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-29T11:20:39.137 DEBUG:teuthology.orchestra.run.smithi176:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:20:39.230 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:39.252+0000 7f7b8b3e1640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-29T11:20:39.230 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:39.253+0000 7f7b8b3e1640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-29T11:20:39.230 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:39.253+0000 7f7b8b3e1640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T11:20:39.230 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:39.253+0000 7f7b8b3e1640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T11:20:39.230 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:39.253+0000 7f7b8b3e1640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T11:20:39.230 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:39.253+0000 7f7b8b3e1640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-29T11:20:42.075 DEBUG:teuthology.orchestra.run.smithi176:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-29T11:20:42.107 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-29T11:20:42.107 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-29T11:20:42.107 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-29T11:20:42.139 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-29T11:20:42.139 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-29T11:20:42.211 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-29T11:20:42.211 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-29T11:20:42.279 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-29T11:20:42.279 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-29T11:20:42.348 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-29T11:20:42.348 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-29T11:20:42.417 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-03-29T11:20:42.417 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-29T11:20:42.449 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-03-29T11:20:42.449 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-29T11:20:42.520 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-03-29T11:20:42.520 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-29T11:20:42.588 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-03-29T11:20:42.588 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-29T11:20:42.658 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-03-29T11:20:42.658 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-29T11:20:42.727 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-29T11:20:42.727 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-29T11:20:42.743 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-03-29T11:20:42.743 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-29T11:20:42.783 INFO:tasks.ceph:Adding keys to all mons... 2024-03-29T11:20:42.783 DEBUG:teuthology.orchestra.run.smithi114:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-29T11:20:42.786 DEBUG:teuthology.orchestra.run.smithi176:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-29T11:20:42.827 INFO:teuthology.orchestra.run.smithi114.stdout:[mgr.y] 2024-03-29T11:20:42.827 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQDrowZm9C6VLBAAdwVMGt94LKI9iEW48/qSqQ== 2024-03-29T11:20:42.827 INFO:teuthology.orchestra.run.smithi114.stdout:[osd.0] 2024-03-29T11:20:42.827 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQDvowZmH5jICxAAxAAMVbNxCy9U9QXdvVBTdA== 2024-03-29T11:20:42.827 INFO:teuthology.orchestra.run.smithi114.stdout:[osd.1] 2024-03-29T11:20:42.827 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQDyowZmuy4GChAA7lNSweLIU2tH6zcDkDMk0Q== 2024-03-29T11:20:42.827 INFO:teuthology.orchestra.run.smithi114.stdout:[osd.2] 2024-03-29T11:20:42.827 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQD1owZmo//CCBAAMBJiDpDZlqUF8xkkSOfK4g== 2024-03-29T11:20:42.827 INFO:teuthology.orchestra.run.smithi114.stdout:[osd.3] 2024-03-29T11:20:42.828 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQD4owZmhOExCBAAe+3TOkNeR6LG3Kv3HHkPVg== 2024-03-29T11:20:42.828 INFO:teuthology.orchestra.run.smithi114.stdout:[mgr.x] 2024-03-29T11:20:42.828 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQDrowZmF1GoMBAA0QPomblcv4pMeO8gSF8Xpg== 2024-03-29T11:20:42.828 INFO:teuthology.orchestra.run.smithi114.stdout:[osd.4] 2024-03-29T11:20:42.828 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQD+owZmAykLExAAaDPWxPxdhdC47XLk3cnk5A== 2024-03-29T11:20:42.828 INFO:teuthology.orchestra.run.smithi114.stdout:[osd.5] 2024-03-29T11:20:42.828 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQABpAZm03HbERAAV9ZfDe0LkWxHRh1jVTxuKA== 2024-03-29T11:20:42.828 INFO:teuthology.orchestra.run.smithi114.stdout:[osd.6] 2024-03-29T11:20:42.829 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQAEpAZmG90/EBAA5J/qmvyy4JSH73wCmKOKgA== 2024-03-29T11:20:42.829 INFO:teuthology.orchestra.run.smithi114.stdout:[osd.7] 2024-03-29T11:20:42.829 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQAHpAZmpRMhDxAAncgdobMX6cIXQsJRQxQcqQ== 2024-03-29T11:20:42.829 INFO:teuthology.orchestra.run.smithi114.stdout:[client.0] 2024-03-29T11:20:42.829 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQDrowZmKX1pMxAA0czHiQkwjtKaZBtPd3/+FQ== 2024-03-29T11:20:42.829 INFO:teuthology.orchestra.run.smithi114.stdout:[client.1] 2024-03-29T11:20:42.829 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQDrowZmZEjVNhAAIEa7jykKLt++lzz2931wKg== 2024-03-29T11:20:42.850 INFO:teuthology.orchestra.run.smithi176.stdout:[mgr.y] 2024-03-29T11:20:42.850 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQDrowZm9C6VLBAAdwVMGt94LKI9iEW48/qSqQ== 2024-03-29T11:20:42.850 INFO:teuthology.orchestra.run.smithi176.stdout:[osd.0] 2024-03-29T11:20:42.850 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQDvowZmH5jICxAAxAAMVbNxCy9U9QXdvVBTdA== 2024-03-29T11:20:42.850 INFO:teuthology.orchestra.run.smithi176.stdout:[osd.1] 2024-03-29T11:20:42.850 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQDyowZmuy4GChAA7lNSweLIU2tH6zcDkDMk0Q== 2024-03-29T11:20:42.850 INFO:teuthology.orchestra.run.smithi176.stdout:[osd.2] 2024-03-29T11:20:42.850 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQD1owZmo//CCBAAMBJiDpDZlqUF8xkkSOfK4g== 2024-03-29T11:20:42.850 INFO:teuthology.orchestra.run.smithi176.stdout:[osd.3] 2024-03-29T11:20:42.850 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQD4owZmhOExCBAAe+3TOkNeR6LG3Kv3HHkPVg== 2024-03-29T11:20:42.850 INFO:teuthology.orchestra.run.smithi176.stdout:[mgr.x] 2024-03-29T11:20:42.850 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQDrowZmF1GoMBAA0QPomblcv4pMeO8gSF8Xpg== 2024-03-29T11:20:42.850 INFO:teuthology.orchestra.run.smithi176.stdout:[osd.4] 2024-03-29T11:20:42.851 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQD+owZmAykLExAAaDPWxPxdhdC47XLk3cnk5A== 2024-03-29T11:20:42.851 INFO:teuthology.orchestra.run.smithi176.stdout:[osd.5] 2024-03-29T11:20:42.851 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQABpAZm03HbERAAV9ZfDe0LkWxHRh1jVTxuKA== 2024-03-29T11:20:42.851 INFO:teuthology.orchestra.run.smithi176.stdout:[osd.6] 2024-03-29T11:20:42.851 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQAEpAZmG90/EBAA5J/qmvyy4JSH73wCmKOKgA== 2024-03-29T11:20:42.851 INFO:teuthology.orchestra.run.smithi176.stdout:[osd.7] 2024-03-29T11:20:42.851 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQAHpAZmpRMhDxAAncgdobMX6cIXQsJRQxQcqQ== 2024-03-29T11:20:42.851 INFO:teuthology.orchestra.run.smithi176.stdout:[client.0] 2024-03-29T11:20:42.851 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQDrowZmKX1pMxAA0czHiQkwjtKaZBtPd3/+FQ== 2024-03-29T11:20:42.851 INFO:teuthology.orchestra.run.smithi176.stdout:[client.1] 2024-03-29T11:20:42.851 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQDrowZmZEjVNhAAIEa7jykKLt++lzz2931wKg== 2024-03-29T11:20:42.852 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:20:42.873 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:20:42.945 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:20:42.948 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:20:43.000 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:20:43.002 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:20:43.053 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:20:43.056 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:20:43.107 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:20:43.110 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:20:43.161 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:20:43.164 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:20:43.215 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:20:43.217 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:20:43.269 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:20:43.271 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:20:43.323 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:20:43.325 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:20:43.376 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:20:43.378 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:20:43.429 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:20:43.432 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:20:43.486 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:20:43.488 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:20:43.543 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-29T11:20:43.543 DEBUG:teuthology.orchestra.run.smithi114:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-29T11:20:43.570 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T11:20:43.667 DEBUG:teuthology.orchestra.run.smithi114:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-29T11:20:43.694 DEBUG:teuthology.orchestra.run.smithi114:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-29T11:20:43.760 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T11:20:43.858 DEBUG:teuthology.orchestra.run.smithi114:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-29T11:20:43.885 DEBUG:teuthology.orchestra.run.smithi176:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-29T11:20:43.913 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T11:20:44.009 DEBUG:teuthology.orchestra.run.smithi176:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-29T11:20:44.036 DEBUG:teuthology.orchestra.run.smithi114:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:20:44.038 DEBUG:teuthology.orchestra.run.smithi176:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:20:44.091 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-29T11:20:44.091 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-29T11:20:44.092 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-03-29T11:20:44.094 INFO:tasks.ceph.mon.a:Started 2024-03-29T11:20:44.094 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-29T11:20:44.094 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-03-29T11:20:44.096 INFO:tasks.ceph.mon.c:Started 2024-03-29T11:20:44.096 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-29T11:20:44.096 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-03-29T11:20:44.134 INFO:tasks.ceph.mon.b:Started 2024-03-29T11:20:44.134 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-29T11:20:44.134 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-29T11:20:44.134 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-03-29T11:20:44.136 INFO:tasks.ceph.mgr.y:Started 2024-03-29T11:20:44.137 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-29T11:20:44.137 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-03-29T11:20:44.139 INFO:tasks.ceph.mgr.x:Started 2024-03-29T11:20:44.139 INFO:tasks.ceph:Setting crush tunables to default 2024-03-29T11:20:44.139 DEBUG:teuthology.orchestra.run.smithi114:> sudo ceph --cluster ceph osd crush tunables default 2024-03-29T11:20:44.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:44.264+0000 7f06d2006640 10 monclient: get_monmap_and_config 2024-03-29T11:20:44.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:44.264+0000 7f06d2006640 10 monclient: build_initial_monmap 2024-03-29T11:20:44.241 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:44.264+0000 7f06d2006640 10 monclient: monmap: 2024-03-29T11:20:44.241 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-03-29T11:20:44.241 INFO:teuthology.orchestra.run.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:20:44.241 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-03-29T11:20:44.265763+0000 2024-03-29T11:20:44.242 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-03-29T11:20:44.265763+0000 2024-03-29T11:20:44.242 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:44.242 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-03-29T11:20:44.242 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:20:44.242 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T11:20:44.242 INFO:teuthology.orchestra.run.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T11:20:44.242 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-03-29T11:20:44.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:44.265+0000 7f06d2006640 1 Processor -- start 2024-03-29T11:20:44.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:44.265+0000 7f06d2006640 1 -- start start 2024-03-29T11:20:44.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:44.265+0000 7f06d2006640 10 monclient: init 2024-03-29T11:20:44.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:44.266+0000 7f06d2006640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:44.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:44.266+0000 7f06d2006640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:20:44.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:44.266+0000 7f06d2006640 10 monclient(hunting): picked mon.noname-a con 0x7f06cc09ac40 addr v1:172.21.15.114:6789/0 2024-03-29T11:20:44.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:44.266+0000 7f06d2006640 10 monclient(hunting): picked mon.noname-c con 0x7f06cc09c870 addr v1:172.21.15.176:6789/0 2024-03-29T11:20:44.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:44.266+0000 7f06d2006640 10 monclient(hunting): picked mon.noname-b con 0x7f06cc0b4360 addr v1:172.21.15.114:6790/0 2024-03-29T11:20:44.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:44.266+0000 7f06d2006640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06cc09dec0 con 0x7f06cc09ac40 2024-03-29T11:20:44.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:44.266+0000 7f06d2006640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06cc09da30 con 0x7f06cc09c870 2024-03-29T11:20:44.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:44.266+0000 7f06d2006640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06cc0b39c0 con 0x7f06cc0b4360 2024-03-29T11:20:44.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:44.266+0000 7f06d2006640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:44.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:44.266+0000 7f06d2006640 10 monclient(hunting): authenticate will time out at 914.448455s 2024-03-29T11:20:44.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:44.266+0000 7f06cb7fe640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f06cc09ac40 0x7f06cc0b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:51980/0 (socket says 172.21.15.114:51980) 2024-03-29T11:20:44.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:44.266+0000 7f06cbfff640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f06cc0b4360 0x7f06cc14d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:39272/0 (socket says 172.21.15.114:39272) 2024-03-29T11:20:44.245 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:44.266+0000 7f06cb7fe640 1 -- 172.21.15.114:0/2762906242 learned_addr learned my addr 172.21.15.114:0/2762906242 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:20:47.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:47.266+0000 7f06c9ffb640 10 monclient(hunting): tick 2024-03-29T11:20:47.243 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:47.266+0000 7f06c9ffb640 10 monclient(hunting): _check_auth_tickets 2024-03-29T11:20:47.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:47.266+0000 7f06c9ffb640 1 monclient(hunting): continuing hunt 2024-03-29T11:20:47.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:47.266+0000 7f06c9ffb640 10 monclient(hunting): _reopen_session rank -1 2024-03-29T11:20:47.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:47.266+0000 7f06c9ffb640 1 -- 172.21.15.114:0/2762906242 >> v1:172.21.15.114:6790/0 conn(0x7f06cc0b4360 legacy=0x7f06cc14d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:47.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:47.266+0000 7f06c9ffb640 1 -- 172.21.15.114:0/2762906242 >> v1:172.21.15.176:6789/0 conn(0x7f06cc09c870 legacy=0x7f06cc13f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:47.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:47.266+0000 7f06c9ffb640 1 -- 172.21.15.114:0/2762906242 >> v1:172.21.15.114:6789/0 conn(0x7f06cc09ac40 legacy=0x7f06cc0b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:47.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:47.266+0000 7f06c9ffb640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:20:47.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:47.266+0000 7f06c9ffb640 10 monclient(hunting): picked mon.noname-b con 0x7f06ac0036e0 addr v1:172.21.15.114:6790/0 2024-03-29T11:20:47.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:47.266+0000 7f06c9ffb640 10 monclient(hunting): picked mon.noname-c con 0x7f06ac007300 addr v1:172.21.15.176:6789/0 2024-03-29T11:20:47.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:47.266+0000 7f06c9ffb640 10 monclient(hunting): picked mon.noname-a con 0x7f06cc09ac40 addr v1:172.21.15.114:6789/0 2024-03-29T11:20:47.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:47.266+0000 7f06c9ffb640 1 -- 172.21.15.114:0/2762906242 --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06cc0b5160 con 0x7f06cc09ac40 2024-03-29T11:20:47.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:47.266+0000 7f06c9ffb640 1 -- 172.21.15.114:0/2762906242 --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06ac00f8d0 con 0x7f06ac007300 2024-03-29T11:20:47.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:47.266+0000 7f06c9ffb640 1 -- 172.21.15.114:0/2762906242 --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06ac010b30 con 0x7f06ac0036e0 2024-03-29T11:20:47.244 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:47.266+0000 7f06c9ffb640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:49.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.435+0000 7f06caffd640 1 -- 172.21.15.114:0/2762906242 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2479738987 0 0) 0x7f06cc0b5160 con 0x7f06cc09ac40 2024-03-29T11:20:49.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.435+0000 7f06caffd640 10 monclient(hunting): _negotiate 2024-03-29T11:20:49.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.435+0000 7f06caffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:49.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.435+0000 7f06caffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:49.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.435+0000 7f06caffd640 10 monclient(hunting): my global_id is 4101 2024-03-29T11:20:49.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.435+0000 7f06caffd640 1 -- 172.21.15.114:0/2762906242 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06b8003520 con 0x7f06cc09ac40 2024-03-29T11:20:49.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.436+0000 7f06caffd640 1 -- 172.21.15.114:0/2762906242 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1140505422 0 0) 0x7f06ac00f8d0 con 0x7f06ac007300 2024-03-29T11:20:49.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.436+0000 7f06caffd640 10 monclient(hunting): _negotiate 2024-03-29T11:20:49.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.436+0000 7f06caffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:49.414 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.436+0000 7f06caffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:49.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.436+0000 7f06caffd640 10 monclient(hunting): my global_id is 4105 2024-03-29T11:20:49.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.436+0000 7f06caffd640 1 -- 172.21.15.114:0/2762906242 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06cc0b5160 con 0x7f06ac007300 2024-03-29T11:20:49.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.436+0000 7f06caffd640 1 -- 172.21.15.114:0/2762906242 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 962203981 0 0) 0x7f06cc0b5160 con 0x7f06ac007300 2024-03-29T11:20:49.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.436+0000 7f06caffd640 1 -- 172.21.15.114:0/2762906242 >> v1:172.21.15.114:6790/0 conn(0x7f06ac0036e0 legacy=0x7f06ac005b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:49.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.436+0000 7f06caffd640 1 -- 172.21.15.114:0/2762906242 >> v1:172.21.15.114:6789/0 conn(0x7f06cc09ac40 legacy=0x7f06ac00d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:49.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.436+0000 7f06caffd640 10 monclient: _finish_hunting 0 2024-03-29T11:20:49.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.436+0000 7f06caffd640 1 monclient: found mon.noname-c 2024-03-29T11:20:49.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.436+0000 7f06caffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:49.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.436+0000 7f06caffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.176:6789/0 2024-03-29T11:20:49.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.436+0000 7f06caffd640 1 -- 172.21.15.114:0/2762906242 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06ac011d90 con 0x7f06ac007300 2024-03-29T11:20:49.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.436+0000 7f06caffd640 10 monclient: _finish_auth 0 2024-03-29T11:20:49.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.436+0000 7f06caffd640 10 monclient: _check_auth_tickets 2024-03-29T11:20:49.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.436+0000 7f06caffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:49.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.436+0000 7f06caffd640 1 -- 172.21.15.114:0/2762906242 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f06b4001690 con 0x7f06ac007300 2024-03-29T11:20:49.415 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.436+0000 7f06caffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:49.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.436+0000 7f06caffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:20:49.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.436+0000 7f06caffd640 10 monclient: dump: 2024-03-29T11:20:49.416 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-03-29T11:20:49.416 INFO:teuthology.orchestra.run.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:20:49.416 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:49.416 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:49.416 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:49.416 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-03-29T11:20:49.416 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:20:49.416 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:20:49.416 INFO:teuthology.orchestra.run.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:20:49.416 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-03-29T11:20:49.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.436+0000 7f06d2006640 5 monclient: authenticate success, global_id 4105 2024-03-29T11:20:49.416 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.436+0000 7f06d2006640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:49.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.437+0000 7f06caffd640 1 -- 172.21.15.114:0/2762906242 <== mon.1 v1:172.21.15.176:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f06b4001860 con 0x7f06ac007300 2024-03-29T11:20:49.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.437+0000 7f06caffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:49.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.437+0000 7f06caffd640 1 -- 172.21.15.114:0/2762906242 <== mon.1 v1:172.21.15.176:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f06b4001bf0 con 0x7f06ac007300 2024-03-29T11:20:49.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.437+0000 7f06caffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:49.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.437+0000 7f06caffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:20:49.417 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.437+0000 7f06caffd640 10 monclient: dump: 2024-03-29T11:20:49.417 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-03-29T11:20:49.417 INFO:teuthology.orchestra.run.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:20:49.417 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:49.417 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:49.417 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:49.417 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-03-29T11:20:49.417 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:20:49.417 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:20:49.417 INFO:teuthology.orchestra.run.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:20:49.418 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-03-29T11:20:49.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.437+0000 7f06d2006640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:49.418 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.437+0000 7f06d2006640 4 set_mon_vals no callback set 2024-03-29T11:20:49.422 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.437+0000 7f06d2006640 10 monclient: shutdown 2024-03-29T11:20:49.422 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.438+0000 7f06d2006640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:49.422 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.438+0000 7f06d2006640 1 -- 172.21.15.114:0/2762906242 >> v1:172.21.15.176:6789/0 conn(0x7f06ac007300 legacy=0x7f06ac009770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:49.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.438+0000 7f06d2006640 1 -- 172.21.15.114:0/2762906242 shutdown_connections 2024-03-29T11:20:49.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.438+0000 7f06d2006640 1 -- 172.21.15.114:0/2762906242 >> 172.21.15.114:0/2762906242 conn(0x7f06cc066b00 msgr2=0x7f06cc066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:49.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.438+0000 7f06d2006640 1 -- 172.21.15.114:0/2762906242 shutdown_connections 2024-03-29T11:20:49.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.438+0000 7f06d2006640 1 -- 172.21.15.114:0/2762906242 wait complete. 2024-03-29T11:20:49.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.438+0000 7f06d2006640 10 monclient: build_initial_monmap 2024-03-29T11:20:49.423 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.438+0000 7f06d2006640 10 monclient: monmap: 2024-03-29T11:20:49.423 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-03-29T11:20:49.423 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:49.423 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-03-29T11:20:49.423 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-03-29T11:20:49.423 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:49.423 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-03-29T11:20:49.423 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:20:49.424 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T11:20:49.424 INFO:teuthology.orchestra.run.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T11:20:49.424 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-03-29T11:20:49.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.439+0000 7f06d2006640 1 librados: starting msgr at 2024-03-29T11:20:49.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.439+0000 7f06d2006640 1 librados: starting objecter 2024-03-29T11:20:49.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.439+0000 7f06d2006640 1 Processor -- start 2024-03-29T11:20:49.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.439+0000 7f06d2006640 1 -- start start 2024-03-29T11:20:49.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.439+0000 7f06d2006640 1 librados: setting wanted keys 2024-03-29T11:20:49.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.439+0000 7f06d2006640 1 librados: calling monclient init 2024-03-29T11:20:49.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.439+0000 7f06d2006640 10 monclient: init 2024-03-29T11:20:49.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.440+0000 7f06d2006640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:49.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.440+0000 7f06d2006640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:20:49.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.440+0000 7f06d2006640 10 monclient(hunting): picked mon.noname-a con 0x7f06ac0036e0 addr v1:172.21.15.114:6789/0 2024-03-29T11:20:49.424 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.440+0000 7f06d2006640 10 monclient(hunting): picked mon.noname-b con 0x7f06ac007300 addr v1:172.21.15.176:6789/0 2024-03-29T11:20:49.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.440+0000 7f06d2006640 10 monclient(hunting): picked mon.noname-c con 0x7f06cc09ac40 addr v1:172.21.15.114:6790/0 2024-03-29T11:20:49.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.440+0000 7f06d2006640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06cc140d50 con 0x7f06ac0036e0 2024-03-29T11:20:49.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.440+0000 7f06cb7fe640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f06ac0036e0 0x7f06cc144540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:52014/0 (socket says 172.21.15.114:52014) 2024-03-29T11:20:49.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.440+0000 7f06cb7fe640 1 -- 172.21.15.114:0/196265604 learned_addr learned my addr 172.21.15.114:0/196265604 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:20:49.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.440+0000 7f06d2006640 1 -- 172.21.15.114:0/196265604 --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06cc145c40 con 0x7f06ac007300 2024-03-29T11:20:49.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.440+0000 7f06d2006640 1 -- 172.21.15.114:0/196265604 --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06cc145f20 con 0x7f06cc09ac40 2024-03-29T11:20:49.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.440+0000 7f06d2006640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:49.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.440+0000 7f06d2006640 10 monclient(hunting): authenticate will time out at 919.622619s 2024-03-29T11:20:49.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.440+0000 7f06c8ff9640 10 client.?.objecter ms_handle_connect 0x7f06ac0036e0 2024-03-29T11:20:49.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.440+0000 7f06c8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:49.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.440+0000 7f06c8ff9640 10 client.?.objecter ms_handle_connect 0x7f06cc09ac40 2024-03-29T11:20:49.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.440+0000 7f06c8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:49.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.440+0000 7f06c8ff9640 10 client.?.objecter ms_handle_connect 0x7f06ac007300 2024-03-29T11:20:49.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.440+0000 7f06c8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:49.425 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.441+0000 7f06c8ff9640 1 -- 172.21.15.114:0/196265604 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1587798372 0 0) 0x7f06cc140d50 con 0x7f06ac0036e0 2024-03-29T11:20:49.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.441+0000 7f06c8ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:20:49.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.441+0000 7f06c8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:49.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.441+0000 7f06c8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:49.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.441+0000 7f06c8ff9640 10 monclient(hunting): my global_id is 4110 2024-03-29T11:20:49.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.441+0000 7f06c8ff9640 1 -- 172.21.15.114:0/196265604 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06ac006a20 con 0x7f06ac0036e0 2024-03-29T11:20:49.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.441+0000 7f06c8ff9640 1 -- 172.21.15.114:0/196265604 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2073373061 0 0) 0x7f06ac006a20 con 0x7f06ac0036e0 2024-03-29T11:20:49.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.441+0000 7f06c8ff9640 1 -- 172.21.15.114:0/196265604 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f06cc140d50 con 0x7f06ac0036e0 2024-03-29T11:20:49.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.441+0000 7f06c8ff9640 1 -- 172.21.15.114:0/196265604 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f06bc001630 con 0x7f06ac0036e0 2024-03-29T11:20:49.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.441+0000 7f06c8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:49.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.441+0000 7f06c8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:20:49.426 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.441+0000 7f06c8ff9640 10 monclient(hunting): dump: 2024-03-29T11:20:49.426 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-03-29T11:20:49.426 INFO:teuthology.orchestra.run.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:20:49.427 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:49.427 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:49.427 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:49.427 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-03-29T11:20:49.427 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:20:49.427 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:20:49.427 INFO:teuthology.orchestra.run.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:20:49.427 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-03-29T11:20:49.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.441+0000 7f06c8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:49.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.442+0000 7f06c8ff9640 1 -- 172.21.15.114:0/196265604 <== mon.0 v1:172.21.15.114:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 239623844 0 0) 0x7f06cc140d50 con 0x7f06ac0036e0 2024-03-29T11:20:49.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.442+0000 7f06c8ff9640 1 -- 172.21.15.114:0/196265604 >> v1:172.21.15.114:6790/0 conn(0x7f06cc09ac40 legacy=0x7f06cc1e1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:49.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.442+0000 7f06c8ff9640 1 -- 172.21.15.114:0/196265604 >> v1:172.21.15.176:6789/0 conn(0x7f06ac007300 legacy=0x7f06cc1d72d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:49.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.442+0000 7f06c8ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:20:49.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.442+0000 7f06c8ff9640 1 monclient: found mon.a 2024-03-29T11:20:49.427 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.442+0000 7f06c8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:49.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.442+0000 7f06c8ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T11:20:49.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.442+0000 7f06c8ff9640 1 -- 172.21.15.114:0/196265604 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06cc143ac0 con 0x7f06ac0036e0 2024-03-29T11:20:49.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.442+0000 7f06c8ff9640 10 monclient: _finish_auth 0 2024-03-29T11:20:49.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.442+0000 7f06c8ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:20:49.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.442+0000 7f06c8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:49.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.442+0000 7f06d2006640 5 monclient: authenticate success, global_id 4110 2024-03-29T11:20:49.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.442+0000 7f06d2006640 10 monclient: _renew_subs 2024-03-29T11:20:49.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.442+0000 7f06d2006640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T11:20:49.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.442+0000 7f06d2006640 1 -- 172.21.15.114:0/196265604 --> v1:172.21.15.114:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06cc143d10 con 0x7f06ac0036e0 2024-03-29T11:20:49.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.442+0000 7f06d2006640 10 client.4110.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:49.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.442+0000 7f06d2006640 10 monclient: _renew_subs 2024-03-29T11:20:49.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.442+0000 7f06d2006640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T11:20:49.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.442+0000 7f06d2006640 1 -- 172.21.15.114:0/196265604 --> v1:172.21.15.114:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f06cc1d7bc0 con 0x7f06ac0036e0 2024-03-29T11:20:49.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.442+0000 7f06d2006640 1 librados: init done 2024-03-29T11:20:49.428 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.442+0000 7f06c8ff9640 1 -- 172.21.15.114:0/196265604 <== mon.0 v1:172.21.15.114:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f06bc001c00 con 0x7f06ac0036e0 2024-03-29T11:20:49.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.442+0000 7f06c8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:49.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.442+0000 7f06c8ff9640 1 -- 172.21.15.114:0/196265604 <== mon.0 v1:172.21.15.114:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f06bc004980 con 0x7f06ac0036e0 2024-03-29T11:20:49.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.442+0000 7f06c8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:49.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.442+0000 7f06c8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:20:49.429 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.442+0000 7f06c8ff9640 10 monclient: dump: 2024-03-29T11:20:49.429 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-03-29T11:20:49.429 INFO:teuthology.orchestra.run.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:20:49.429 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:49.429 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:49.429 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:49.429 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-03-29T11:20:49.429 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:20:49.429 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:20:49.429 INFO:teuthology.orchestra.run.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:20:49.429 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-03-29T11:20:49.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.442+0000 7f06ca7fc640 4 set_mon_vals no callback set 2024-03-29T11:20:49.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.442+0000 7f06c8ff9640 1 -- 172.21.15.114:0/196265604 <== mon.0 v1:172.21.15.114:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f06bc004f30 con 0x7f06ac0036e0 2024-03-29T11:20:49.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.442+0000 7f06c8ff9640 1 -- 172.21.15.114:0/196265604 <== mon.0 v1:172.21.15.114:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 1399776776 0 0) 0x7f06bc006330 con 0x7f06ac0036e0 2024-03-29T11:20:49.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.442+0000 7f06c8ff9640 10 client.4110.objecter ms_dispatch 0x7f06cc000b90 osd_map(1..1 src has 1..1) v4 2024-03-29T11:20:49.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.442+0000 7f06c8ff9640 3 client.4110.objecter handle_osd_map got epochs [1,1] > 0 2024-03-29T11:20:49.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.442+0000 7f06c8ff9640 3 client.4110.objecter handle_osd_map decoding full epoch 1 2024-03-29T11:20:49.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.442+0000 7f06c8ff9640 20 client.4110.objecter dump_active .. 0 homeless 2024-03-29T11:20:49.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.444+0000 7f06d2006640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:49.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.444+0000 7f06d2006640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:49.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.444+0000 7f06d2006640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T11:20:49.430 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.444+0000 7f06d2006640 1 -- 172.21.15.114:0/196265604 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06cc09b020 con 0x7f06ac0036e0 2024-03-29T11:20:49.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.447+0000 7f06c8ff9640 1 -- 172.21.15.114:0/196265604 <== mon.0 v1:172.21.15.114:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f06bc0053c0 con 0x7f06ac0036e0 2024-03-29T11:20:49.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.447+0000 7f06c8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:49.431 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.447+0000 7f06c8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:49.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.541+0000 7f06d2006640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T11:20:49.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.541+0000 7f06d2006640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T11:20:49.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.541+0000 7f06d2006640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T11:20:49.518 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.541+0000 7f06d2006640 1 -- 172.21.15.114:0/196265604 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f06cc04f730 con 0x7f06ac0036e0 2024-03-29T11:20:49.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.543+0000 7f06c8ff9640 1 -- 172.21.15.114:0/196265604 <== mon.0 v1:172.21.15.114:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (unknown 2277712955 0 0) 0x7f06bc0055e0 con 0x7f06ac0036e0 2024-03-29T11:20:49.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.543+0000 7f06c8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T11:20:49.520 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.543+0000 7f06c8ff9640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-03-29T11:20:49.520 INFO:teuthology.orchestra.run.smithi114.stderr:adjusted tunables profile to default 2024-03-29T11:20:49.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.544+0000 7f06d2006640 10 librados: watch_flush enter 2024-03-29T11:20:49.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.544+0000 7f06d2006640 10 librados: watch_flush exit 2024-03-29T11:20:49.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.544+0000 7f06d2006640 20 client.4110.objecter shutdown clearing up homeless session... 2024-03-29T11:20:49.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.544+0000 7f06d2006640 10 client.4110.objecter successfully canceled tick 2024-03-29T11:20:49.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.544+0000 7f06d2006640 10 monclient: shutdown 2024-03-29T11:20:49.521 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.544+0000 7f06d2006640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:49.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.544+0000 7f06d2006640 1 -- 172.21.15.114:0/196265604 >> v1:172.21.15.114:6789/0 conn(0x7f06ac0036e0 legacy=0x7f06cc144540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:49.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.544+0000 7f06d2006640 1 -- 172.21.15.114:0/196265604 shutdown_connections 2024-03-29T11:20:49.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.544+0000 7f06d2006640 1 -- 172.21.15.114:0/196265604 >> 172.21.15.114:0/196265604 conn(0x7f06cc066b00 msgr2=0x7f06cc066ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:49.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.544+0000 7f06d2006640 1 -- 172.21.15.114:0/196265604 shutdown_connections 2024-03-29T11:20:49.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.544+0000 7f06d2006640 1 -- 172.21.15.114:0/196265604 wait complete. 2024-03-29T11:20:49.522 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:49.544+0000 7f06d2006640 1 librados: shutdown 2024-03-29T11:20:49.532 INFO:tasks.ceph:check_enable_crimson: False 2024-03-29T11:20:49.532 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-29T11:20:49.532 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-29T11:20:49.532 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-29T11:20:49.559 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-29T11:20:49.559 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-29T11:20:49.626 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-29T11:20:49.626 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-29T11:20:49.693 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2024-03-29T11:20:49.693 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-29T11:20:49.760 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-03-29T11:20:49.760 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-29T11:20:49.787 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-03-29T11:20:49.788 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-29T11:20:49.854 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-03-29T11:20:49.854 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-29T11:20:49.899 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-03-29T11:20:49.923+0000 7fedaa420040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T11:20:49.905 INFO:tasks.ceph.mgr.y.smithi114.stderr:2024-03-29T11:20:49.929+0000 7ffbece52040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T11:20:49.922 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-03-29T11:20:49.922 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-29T11:20:49.956 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-03-29T11:20:49.979+0000 7fedaa420040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T11:20:49.959 INFO:tasks.ceph.mgr.y.smithi114.stderr:2024-03-29T11:20:49.983+0000 7ffbece52040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T11:20:49.982 DEBUG:teuthology.orchestra.run.smithi176:> sudo ceph --cluster ceph osd new a64785d3-6b11-499a-b706-cc827ab96a74 0 2024-03-29T11:20:50.123 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.146+0000 7fe42ea0f640 10 monclient: get_monmap_and_config 2024-03-29T11:20:50.123 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.146+0000 7fe42ea0f640 10 monclient: build_initial_monmap 2024-03-29T11:20:50.123 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.146+0000 7fe42ea0f640 10 monclient: monmap: 2024-03-29T11:20:50.123 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 0 2024-03-29T11:20:50.123 INFO:teuthology.orchestra.run.smithi176.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:20:50.124 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-29T11:20:50.147184+0000 2024-03-29T11:20:50.124 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-29T11:20:50.147184+0000 2024-03-29T11:20:50.124 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:50.124 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T11:20:50.124 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:20:50.124 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T11:20:50.124 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T11:20:50.124 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T11:20:50.124 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.146+0000 7fe42ea0f640 1 Processor -- start 2024-03-29T11:20:50.124 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.146+0000 7fe42ea0f640 1 -- start start 2024-03-29T11:20:50.124 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.146+0000 7fe42ea0f640 10 monclient: init 2024-03-29T11:20:50.135 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.147+0000 7fe42ea0f640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:50.135 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.147+0000 7fe42ea0f640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:20:50.136 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.147+0000 7fe42ea0f640 10 monclient(hunting): picked mon.noname-c con 0x7fe42809ac40 addr v1:172.21.15.176:6789/0 2024-03-29T11:20:50.136 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.147+0000 7fe42ea0f640 10 monclient(hunting): picked mon.noname-a con 0x7fe42809c870 addr v1:172.21.15.114:6789/0 2024-03-29T11:20:50.136 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.147+0000 7fe42ea0f640 10 monclient(hunting): picked mon.noname-b con 0x7fe4280b4360 addr v1:172.21.15.114:6790/0 2024-03-29T11:20:50.136 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.147+0000 7fe42ea0f640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe42809dec0 con 0x7fe42809c870 2024-03-29T11:20:50.136 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.147+0000 7fe42ea0f640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe42809da30 con 0x7fe42809ac40 2024-03-29T11:20:50.136 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.147+0000 7fe42ea0f640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4280b39c0 con 0x7fe4280b4360 2024-03-29T11:20:50.136 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.147+0000 7fe42ea0f640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:50.136 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.147+0000 7fe42ea0f640 10 monclient(hunting): authenticate will time out at 915.488666s 2024-03-29T11:20:50.136 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.147+0000 7fe427fff640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7fe42809ac40 0x7fe4280b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.176:48996/0 (socket says 172.21.15.176:48996) 2024-03-29T11:20:50.136 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.147+0000 7fe427fff640 1 -- 172.21.15.176:0/4049673630 learned_addr learned my addr 172.21.15.176:0/4049673630 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-29T11:20:50.136 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.147+0000 7fe426ffd640 1 -- 172.21.15.176:0/4049673630 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1204491961 0 0) 0x7fe42809da30 con 0x7fe42809ac40 2024-03-29T11:20:50.137 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.147+0000 7fe426ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:20:50.137 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.147+0000 7fe426ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:50.137 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.147+0000 7fe426ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:50.137 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.147+0000 7fe426ffd640 10 monclient(hunting): my global_id is 4108 2024-03-29T11:20:50.137 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.147+0000 7fe426ffd640 1 -- 172.21.15.176:0/4049673630 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe410003520 con 0x7fe42809ac40 2024-03-29T11:20:50.137 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.147+0000 7fe426ffd640 1 -- 172.21.15.176:0/4049673630 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1660471991 0 0) 0x7fe418001040 con 0x7fe42809c870 2024-03-29T11:20:50.137 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.148+0000 7fe426ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:20:50.137 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.148+0000 7fe426ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:50.137 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.148+0000 7fe426ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:50.137 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.148+0000 7fe426ffd640 10 monclient(hunting): my global_id is 4113 2024-03-29T11:20:50.137 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.148+0000 7fe426ffd640 1 -- 172.21.15.176:0/4049673630 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe42809da30 con 0x7fe42809c870 2024-03-29T11:20:50.138 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.148+0000 7fe426ffd640 1 -- 172.21.15.176:0/4049673630 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3248528013 0 0) 0x7fe4280b39c0 con 0x7fe4280b4360 2024-03-29T11:20:50.138 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.148+0000 7fe426ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:20:50.138 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.148+0000 7fe426ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:50.138 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.148+0000 7fe426ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:50.138 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.148+0000 7fe426ffd640 10 monclient(hunting): my global_id is 4103 2024-03-29T11:20:50.138 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.148+0000 7fe426ffd640 1 -- 172.21.15.176:0/4049673630 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe418001040 con 0x7fe4280b4360 2024-03-29T11:20:50.138 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.148+0000 7fe426ffd640 1 -- 172.21.15.176:0/4049673630 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1833009122 0 0) 0x7fe410003520 con 0x7fe42809ac40 2024-03-29T11:20:50.138 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.148+0000 7fe426ffd640 1 -- 172.21.15.176:0/4049673630 >> v1:172.21.15.114:6790/0 conn(0x7fe4280b4360 legacy=0x7fe42814d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:50.138 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.148+0000 7fe426ffd640 1 -- 172.21.15.176:0/4049673630 >> v1:172.21.15.114:6789/0 conn(0x7fe42809c870 legacy=0x7fe42813f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:50.138 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.148+0000 7fe426ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:20:50.139 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.148+0000 7fe426ffd640 1 monclient: found mon.noname-c 2024-03-29T11:20:50.139 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.148+0000 7fe426ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:50.139 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.148+0000 7fe426ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.176:6789/0 2024-03-29T11:20:50.139 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.148+0000 7fe426ffd640 1 -- 172.21.15.176:0/4049673630 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4280b5160 con 0x7fe42809ac40 2024-03-29T11:20:50.139 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.148+0000 7fe426ffd640 10 monclient: _finish_auth 0 2024-03-29T11:20:50.139 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.148+0000 7fe426ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:20:50.139 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.148+0000 7fe426ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:50.139 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.148+0000 7fe426ffd640 1 -- 172.21.15.176:0/4049673630 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fe41c001610 con 0x7fe42809ac40 2024-03-29T11:20:50.139 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.148+0000 7fe42ea0f640 5 monclient: authenticate success, global_id 4108 2024-03-29T11:20:50.139 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.148+0000 7fe42ea0f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:50.139 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.148+0000 7fe426ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:50.140 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.148+0000 7fe426ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:20:50.140 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.148+0000 7fe426ffd640 10 monclient: dump: 2024-03-29T11:20:50.140 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-03-29T11:20:50.140 INFO:teuthology.orchestra.run.smithi176.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:20:50.140 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:50.140 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:50.140 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:50.140 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T11:20:50.140 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:20:50.140 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:20:50.140 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:20:50.140 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T11:20:50.140 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.148+0000 7fe42ea0f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:50.141 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.148+0000 7fe426ffd640 1 -- 172.21.15.176:0/4049673630 <== mon.1 v1:172.21.15.176:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe41c001770 con 0x7fe42809ac40 2024-03-29T11:20:50.141 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.148+0000 7fe426ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:50.141 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.148+0000 7fe426ffd640 1 -- 172.21.15.176:0/4049673630 <== mon.1 v1:172.21.15.176:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fe41c001bc0 con 0x7fe42809ac40 2024-03-29T11:20:50.141 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.148+0000 7fe42ea0f640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:50.141 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.148+0000 7fe42ea0f640 4 set_mon_vals no callback set 2024-03-29T11:20:50.141 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.148+0000 7fe426ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:50.141 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.148+0000 7fe426ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:20:50.141 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.148+0000 7fe426ffd640 10 monclient: dump: 2024-03-29T11:20:50.141 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-03-29T11:20:50.141 INFO:teuthology.orchestra.run.smithi176.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:20:50.141 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:50.142 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:50.142 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:50.142 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T11:20:50.142 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:20:50.142 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:20:50.142 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:20:50.142 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T11:20:50.142 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.148+0000 7fe42ea0f640 10 monclient: shutdown 2024-03-29T11:20:50.142 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.148+0000 7fe42ea0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:50.142 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.148+0000 7fe42ea0f640 1 -- 172.21.15.176:0/4049673630 >> v1:172.21.15.176:6789/0 conn(0x7fe42809ac40 legacy=0x7fe4280b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:50.143 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.148+0000 7fe42ea0f640 1 -- 172.21.15.176:0/4049673630 shutdown_connections 2024-03-29T11:20:50.143 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.148+0000 7fe42ea0f640 1 -- 172.21.15.176:0/4049673630 >> 172.21.15.176:0/4049673630 conn(0x7fe428066b00 msgr2=0x7fe428066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:50.143 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.148+0000 7fe42ea0f640 1 -- 172.21.15.176:0/4049673630 shutdown_connections 2024-03-29T11:20:50.143 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.148+0000 7fe42ea0f640 1 -- 172.21.15.176:0/4049673630 wait complete. 2024-03-29T11:20:50.143 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.149+0000 7fe42ea0f640 10 monclient: build_initial_monmap 2024-03-29T11:20:50.143 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.149+0000 7fe42ea0f640 10 monclient: monmap: 2024-03-29T11:20:50.143 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 0 2024-03-29T11:20:50.143 INFO:teuthology.orchestra.run.smithi176.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:50.143 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 0.000000 2024-03-29T11:20:50.143 INFO:teuthology.orchestra.run.smithi176.stderr:created 0.000000 2024-03-29T11:20:50.143 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:50.143 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T11:20:50.144 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:20:50.144 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T11:20:50.144 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T11:20:50.144 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T11:20:50.144 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.149+0000 7fe42ea0f640 1 librados: starting msgr at 2024-03-29T11:20:50.144 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.149+0000 7fe42ea0f640 1 librados: starting objecter 2024-03-29T11:20:50.144 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.149+0000 7fe42ea0f640 1 Processor -- start 2024-03-29T11:20:50.144 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.149+0000 7fe42ea0f640 1 -- start start 2024-03-29T11:20:50.145 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.149+0000 7fe42ea0f640 1 librados: setting wanted keys 2024-03-29T11:20:50.145 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.149+0000 7fe42ea0f640 1 librados: calling monclient init 2024-03-29T11:20:50.145 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.149+0000 7fe42ea0f640 10 monclient: init 2024-03-29T11:20:50.145 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.150+0000 7fe42ea0f640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:50.145 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.150+0000 7fe42ea0f640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:20:50.145 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.150+0000 7fe42ea0f640 10 monclient(hunting): picked mon.noname-b con 0x7fe42809ac40 addr v1:172.21.15.176:6789/0 2024-03-29T11:20:50.145 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.150+0000 7fe42ea0f640 10 monclient(hunting): picked mon.noname-c con 0x7fe4280b69a0 addr v1:172.21.15.114:6790/0 2024-03-29T11:20:50.145 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.150+0000 7fe42ea0f640 10 monclient(hunting): picked mon.noname-a con 0x7fe4280b18a0 addr v1:172.21.15.114:6789/0 2024-03-29T11:20:50.145 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.150+0000 7fe42ea0f640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4281db320 con 0x7fe4280b18a0 2024-03-29T11:20:50.145 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.150+0000 7fe42ea0f640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4281da800 con 0x7fe42809ac40 2024-03-29T11:20:50.146 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.150+0000 7fe42ea0f640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4281da990 con 0x7fe4280b69a0 2024-03-29T11:20:50.146 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.150+0000 7fe42ea0f640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:50.146 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.150+0000 7fe42ea0f640 10 monclient(hunting): authenticate will time out at 915.491678s 2024-03-29T11:20:50.146 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.150+0000 7fe427fff640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7fe42809ac40 0x7fe4280b6290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.176:49004/0 (socket says 172.21.15.176:49004) 2024-03-29T11:20:50.146 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.150+0000 7fe427fff640 1 -- 172.21.15.176:0/4109458886 learned_addr learned my addr 172.21.15.176:0/4109458886 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-29T11:20:50.146 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.150+0000 7fe424ff9640 10 client.?.objecter ms_handle_connect 0x7fe42809ac40 2024-03-29T11:20:50.147 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.150+0000 7fe424ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:50.147 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.150+0000 7fe424ff9640 10 client.?.objecter ms_handle_connect 0x7fe4280b69a0 2024-03-29T11:20:50.147 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.150+0000 7fe424ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:50.147 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.150+0000 7fe424ff9640 10 client.?.objecter ms_handle_connect 0x7fe4280b18a0 2024-03-29T11:20:50.147 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.150+0000 7fe424ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:50.147 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.150+0000 7fe424ff9640 1 -- 172.21.15.176:0/4109458886 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2086078206 0 0) 0x7fe4281da800 con 0x7fe42809ac40 2024-03-29T11:20:50.147 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.150+0000 7fe424ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:20:50.147 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.150+0000 7fe424ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:50.147 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.150+0000 7fe424ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:50.148 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.150+0000 7fe424ff9640 10 monclient(hunting): my global_id is 4111 2024-03-29T11:20:50.148 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.150+0000 7fe424ff9640 1 -- 172.21.15.176:0/4109458886 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe414004eb0 con 0x7fe42809ac40 2024-03-29T11:20:50.148 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.150+0000 7fe424ff9640 1 -- 172.21.15.176:0/4109458886 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2104537212 0 0) 0x7fe4281da990 con 0x7fe4280b69a0 2024-03-29T11:20:50.148 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.150+0000 7fe424ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:20:50.148 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.150+0000 7fe424ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:50.149 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.150+0000 7fe424ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:50.149 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.150+0000 7fe424ff9640 10 monclient(hunting): my global_id is 4106 2024-03-29T11:20:50.149 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.150+0000 7fe424ff9640 1 -- 172.21.15.176:0/4109458886 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4281da800 con 0x7fe4280b69a0 2024-03-29T11:20:50.149 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.150+0000 7fe424ff9640 1 -- 172.21.15.176:0/4109458886 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2453913556 0 0) 0x7fe4281db320 con 0x7fe4280b18a0 2024-03-29T11:20:50.149 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.150+0000 7fe424ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:20:50.149 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.150+0000 7fe424ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:50.149 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.150+0000 7fe424ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:50.149 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.150+0000 7fe424ff9640 10 monclient(hunting): my global_id is 4116 2024-03-29T11:20:50.150 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.150+0000 7fe424ff9640 1 -- 172.21.15.176:0/4109458886 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4281da990 con 0x7fe4280b18a0 2024-03-29T11:20:50.150 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.151+0000 7fe424ff9640 1 -- 172.21.15.176:0/4109458886 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 524161565 0 0) 0x7fe414004eb0 con 0x7fe42809ac40 2024-03-29T11:20:50.150 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.151+0000 7fe424ff9640 1 -- 172.21.15.176:0/4109458886 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe4281db320 con 0x7fe42809ac40 2024-03-29T11:20:50.150 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.151+0000 7fe424ff9640 1 -- 172.21.15.176:0/4109458886 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fe41c001b40 con 0x7fe42809ac40 2024-03-29T11:20:50.150 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.151+0000 7fe424ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:50.150 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.151+0000 7fe424ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:20:50.150 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.151+0000 7fe424ff9640 10 monclient(hunting): dump: 2024-03-29T11:20:50.150 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-03-29T11:20:50.150 INFO:teuthology.orchestra.run.smithi176.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:20:50.150 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:50.150 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:50.151 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:50.151 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T11:20:50.151 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:20:50.151 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:20:50.151 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:20:50.151 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T11:20:50.151 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.151+0000 7fe424ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:50.151 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.151+0000 7fe424ff9640 1 -- 172.21.15.176:0/4109458886 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 465540022 0 0) 0x7fe4281da800 con 0x7fe4280b69a0 2024-03-29T11:20:50.151 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.151+0000 7fe424ff9640 1 -- 172.21.15.176:0/4109458886 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe414004eb0 con 0x7fe4280b69a0 2024-03-29T11:20:50.152 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.151+0000 7fe424ff9640 1 -- 172.21.15.176:0/4109458886 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3192049704 0 0) 0x7fe4281da990 con 0x7fe4280b18a0 2024-03-29T11:20:50.152 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.151+0000 7fe424ff9640 1 -- 172.21.15.176:0/4109458886 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe4281da800 con 0x7fe4280b18a0 2024-03-29T11:20:50.152 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.151+0000 7fe424ff9640 1 -- 172.21.15.176:0/4109458886 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fe4180017d0 con 0x7fe4280b69a0 2024-03-29T11:20:50.152 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.151+0000 7fe424ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:50.152 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.151+0000 7fe424ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:20:50.152 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.151+0000 7fe424ff9640 10 monclient(hunting): dump: 2024-03-29T11:20:50.152 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-03-29T11:20:50.152 INFO:teuthology.orchestra.run.smithi176.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:20:50.152 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:50.152 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:50.152 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:50.153 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T11:20:50.153 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:20:50.153 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:20:50.153 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:20:50.153 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T11:20:50.153 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.151+0000 7fe424ff9640 1 -- 172.21.15.176:0/4109458886 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fe420001a10 con 0x7fe4280b18a0 2024-03-29T11:20:50.153 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.151+0000 7fe424ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:50.153 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.151+0000 7fe424ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:20:50.153 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.151+0000 7fe424ff9640 10 monclient(hunting): dump: 2024-03-29T11:20:50.153 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-03-29T11:20:50.153 INFO:teuthology.orchestra.run.smithi176.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:20:50.153 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:50.153 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:50.154 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:50.154 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T11:20:50.154 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:20:50.154 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:20:50.154 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:20:50.154 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T11:20:50.154 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.151+0000 7fe424ff9640 1 -- 172.21.15.176:0/4109458886 <== mon.1 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2140219993 0 0) 0x7fe4281db320 con 0x7fe42809ac40 2024-03-29T11:20:50.154 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.151+0000 7fe424ff9640 1 -- 172.21.15.176:0/4109458886 >> v1:172.21.15.114:6790/0 conn(0x7fe4280b69a0 legacy=0x7fe428140cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:50.154 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.151+0000 7fe424ff9640 1 -- 172.21.15.176:0/4109458886 >> v1:172.21.15.114:6789/0 conn(0x7fe4280b18a0 legacy=0x7fe4281e1ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:50.154 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.151+0000 7fe424ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:20:50.154 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.151+0000 7fe424ff9640 1 monclient: found mon.b 2024-03-29T11:20:50.154 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.151+0000 7fe424ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:50.155 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.151+0000 7fe424ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:20:50.155 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.151+0000 7fe424ff9640 1 -- 172.21.15.176:0/4109458886 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4281dab20 con 0x7fe42809ac40 2024-03-29T11:20:50.155 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.151+0000 7fe424ff9640 10 monclient: _finish_auth 0 2024-03-29T11:20:50.155 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.151+0000 7fe424ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:20:50.155 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.151+0000 7fe424ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:50.155 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.151+0000 7fe42ea0f640 5 monclient: authenticate success, global_id 4111 2024-03-29T11:20:50.155 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.151+0000 7fe42ea0f640 10 monclient: _renew_subs 2024-03-29T11:20:50.155 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.151+0000 7fe42ea0f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:20:50.155 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.151+0000 7fe42ea0f640 1 -- 172.21.15.176:0/4109458886 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4281e32c0 con 0x7fe42809ac40 2024-03-29T11:20:50.155 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.151+0000 7fe42ea0f640 10 client.4111.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:50.155 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.151+0000 7fe42ea0f640 10 monclient: _renew_subs 2024-03-29T11:20:50.155 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.151+0000 7fe42ea0f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:20:50.156 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.151+0000 7fe42ea0f640 1 -- 172.21.15.176:0/4109458886 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4281e3630 con 0x7fe42809ac40 2024-03-29T11:20:50.156 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.151+0000 7fe42ea0f640 1 librados: init done 2024-03-29T11:20:50.156 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.151+0000 7fe424ff9640 1 -- 172.21.15.176:0/4109458886 <== mon.1 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe41c004a90 con 0x7fe42809ac40 2024-03-29T11:20:50.156 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.151+0000 7fe424ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:50.156 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.151+0000 7fe424ff9640 1 -- 172.21.15.176:0/4109458886 <== mon.1 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fe41c004de0 con 0x7fe42809ac40 2024-03-29T11:20:50.156 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.151+0000 7fe424ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:50.156 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.151+0000 7fe424ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:20:50.156 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.151+0000 7fe4267fc640 4 set_mon_vals no callback set 2024-03-29T11:20:50.156 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.151+0000 7fe424ff9640 10 monclient: dump: 2024-03-29T11:20:50.156 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-03-29T11:20:50.156 INFO:teuthology.orchestra.run.smithi176.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:20:50.156 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:50.156 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:50.157 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:50.157 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T11:20:50.157 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:20:50.157 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:20:50.157 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:20:50.157 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T11:20:50.157 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.151+0000 7fe424ff9640 1 -- 172.21.15.176:0/4109458886 <== mon.1 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fe41c0052d0 con 0x7fe42809ac40 2024-03-29T11:20:50.157 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.151+0000 7fe424ff9640 1 -- 172.21.15.176:0/4109458886 <== mon.1 v1:172.21.15.176:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (unknown 852140446 0 0) 0x7fe41c0068e0 con 0x7fe42809ac40 2024-03-29T11:20:50.157 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.151+0000 7fe424ff9640 10 client.4111.objecter ms_dispatch 0x7fe428000b90 osd_map(2..2 src has 1..2) v4 2024-03-29T11:20:50.157 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.151+0000 7fe424ff9640 3 client.4111.objecter handle_osd_map got epochs [2,2] > 0 2024-03-29T11:20:50.157 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.151+0000 7fe424ff9640 3 client.4111.objecter handle_osd_map decoding full epoch 2 2024-03-29T11:20:50.158 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.151+0000 7fe424ff9640 20 client.4111.objecter dump_active .. 0 homeless 2024-03-29T11:20:50.158 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.152+0000 7fe42ea0f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:50.158 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.152+0000 7fe42ea0f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:50.158 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.152+0000 7fe42ea0f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:20:50.158 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.152+0000 7fe42ea0f640 1 -- 172.21.15.176:0/4109458886 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe408005080 con 0x7fe42809ac40 2024-03-29T11:20:50.158 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.154+0000 7fe424ff9640 1 -- 172.21.15.176:0/4109458886 <== mon.1 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fe41c006aa0 con 0x7fe42809ac40 2024-03-29T11:20:50.158 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.154+0000 7fe424ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:50.158 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.154+0000 7fe424ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:50.231 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.255+0000 7fe42ea0f640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "a64785d3-6b11-499a-b706-cc827ab96a74", "id": 0}] 2024-03-29T11:20:50.232 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.255+0000 7fe42ea0f640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "a64785d3-6b11-499a-b706-cc827ab96a74", "id": 0}] 2024-03-29T11:20:50.232 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.255+0000 7fe42ea0f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:20:50.232 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.255+0000 7fe42ea0f640 1 -- 172.21.15.176:0/4109458886 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd new", "uuid": "a64785d3-6b11-499a-b706-cc827ab96a74", "id": 0} v 0) v1 -- 0x7fe4080058b0 con 0x7fe42809ac40 2024-03-29T11:20:50.235 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.258+0000 7fe424ff9640 1 -- 172.21.15.176:0/4109458886 <== mon.1 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "a64785d3-6b11-499a-b706-cc827ab96a74", "id": 0}]=0 v3) v1 ==== 112+0+2 (unknown 3894161557 0 732707515) 0x7fe41c0058e0 con 0x7fe42809ac40 2024-03-29T11:20:50.235 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.258+0000 7fe424ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "a64785d3-6b11-499a-b706-cc827ab96a74", "id": 0}] 2024-03-29T11:20:50.235 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.258+0000 7fe424ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:50.235 INFO:teuthology.orchestra.run.smithi176.stdout:0 2024-03-29T11:20:50.237 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.259+0000 7fe42ea0f640 10 librados: watch_flush enter 2024-03-29T11:20:50.237 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.259+0000 7fe42ea0f640 10 librados: watch_flush exit 2024-03-29T11:20:50.237 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.259+0000 7fe42ea0f640 20 client.4111.objecter shutdown clearing up homeless session... 2024-03-29T11:20:50.237 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.259+0000 7fe42ea0f640 10 client.4111.objecter successfully canceled tick 2024-03-29T11:20:50.237 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.259+0000 7fe42ea0f640 10 monclient: shutdown 2024-03-29T11:20:50.237 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.259+0000 7fe42ea0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:50.237 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.259+0000 7fe42ea0f640 1 -- 172.21.15.176:0/4109458886 >> v1:172.21.15.176:6789/0 conn(0x7fe42809ac40 legacy=0x7fe4280b6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:50.237 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.259+0000 7fe42ea0f640 1 -- 172.21.15.176:0/4109458886 shutdown_connections 2024-03-29T11:20:50.238 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.259+0000 7fe42ea0f640 1 -- 172.21.15.176:0/4109458886 >> 172.21.15.176:0/4109458886 conn(0x7fe428066b00 msgr2=0x7fe42813e870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:50.238 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.259+0000 7fe42ea0f640 1 -- 172.21.15.176:0/4109458886 shutdown_connections 2024-03-29T11:20:50.238 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.259+0000 7fe42ea0f640 1 -- 172.21.15.176:0/4109458886 wait complete. 2024-03-29T11:20:50.238 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.259+0000 7fe42ea0f640 1 librados: shutdown 2024-03-29T11:20:50.246 DEBUG:teuthology.orchestra.run.smithi176:> sudo ceph --cluster ceph osd new 5882dadb-e7a8-4604-ba9a-418013c106c8 1 2024-03-29T11:20:50.345 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.369+0000 7f283dbd6640 10 monclient: get_monmap_and_config 2024-03-29T11:20:50.346 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.369+0000 7f283dbd6640 10 monclient: build_initial_monmap 2024-03-29T11:20:50.346 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.369+0000 7f283dbd6640 10 monclient: monmap: 2024-03-29T11:20:50.346 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 0 2024-03-29T11:20:50.346 INFO:teuthology.orchestra.run.smithi176.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:20:50.346 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-29T11:20:50.370259+0000 2024-03-29T11:20:50.346 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-29T11:20:50.370259+0000 2024-03-29T11:20:50.346 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:50.346 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T11:20:50.346 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:20:50.346 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T11:20:50.346 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T11:20:50.347 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T11:20:50.356 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.369+0000 7f283dbd6640 1 Processor -- start 2024-03-29T11:20:50.356 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.369+0000 7f283dbd6640 1 -- start start 2024-03-29T11:20:50.356 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.369+0000 7f283dbd6640 10 monclient: init 2024-03-29T11:20:50.356 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.370+0000 7f283dbd6640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:50.356 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.370+0000 7f283dbd6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:20:50.356 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.370+0000 7f283dbd6640 10 monclient(hunting): picked mon.noname-c con 0x7f2838066b90 addr v1:172.21.15.176:6789/0 2024-03-29T11:20:50.356 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.370+0000 7f283dbd6640 10 monclient(hunting): picked mon.noname-b con 0x7f283813fe30 addr v1:172.21.15.114:6790/0 2024-03-29T11:20:50.356 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.370+0000 7f283dbd6640 10 monclient(hunting): picked mon.noname-a con 0x7f283813d0f0 addr v1:172.21.15.114:6789/0 2024-03-29T11:20:50.356 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.370+0000 7f283dbd6640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28381403a0 con 0x7f283813d0f0 2024-03-29T11:20:50.356 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.370+0000 7f283dbd6640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28380b0630 con 0x7f2838066b90 2024-03-29T11:20:50.356 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.370+0000 7f283dbd6640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28380b0910 con 0x7f283813fe30 2024-03-29T11:20:50.356 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.370+0000 7f283dbd6640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:50.356 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.370+0000 7f283dbd6640 10 monclient(hunting): authenticate will time out at 915.711790s 2024-03-29T11:20:50.357 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.370+0000 7f28377fe640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f2838066b90 0x7f2838140830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.176:49014/0 (socket says 172.21.15.176:49014) 2024-03-29T11:20:50.357 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.370+0000 7f28377fe640 1 -- 172.21.15.176:0/2513601159 learned_addr learned my addr 172.21.15.176:0/2513601159 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-29T11:20:50.357 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.370+0000 7f28367fc640 1 -- 172.21.15.176:0/2513601159 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1729503247 0 0) 0x7f28380b0630 con 0x7f2838066b90 2024-03-29T11:20:50.357 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.370+0000 7f28367fc640 10 monclient(hunting): _negotiate 2024-03-29T11:20:50.357 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.370+0000 7f28367fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:50.357 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.370+0000 7f28367fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:50.357 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.370+0000 7f28367fc640 10 monclient(hunting): my global_id is 4114 2024-03-29T11:20:50.357 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.370+0000 7f28367fc640 1 -- 172.21.15.176:0/2513601159 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2820003520 con 0x7f2838066b90 2024-03-29T11:20:50.357 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.371+0000 7f28367fc640 1 -- 172.21.15.176:0/2513601159 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 993569940 0 0) 0x7f28381403a0 con 0x7f283813d0f0 2024-03-29T11:20:50.357 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.371+0000 7f28367fc640 10 monclient(hunting): _negotiate 2024-03-29T11:20:50.357 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.371+0000 7f28367fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:50.357 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.371+0000 7f28367fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:50.357 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.371+0000 7f28367fc640 10 monclient(hunting): my global_id is 4119 2024-03-29T11:20:50.357 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.371+0000 7f28367fc640 1 -- 172.21.15.176:0/2513601159 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28380b0630 con 0x7f283813d0f0 2024-03-29T11:20:50.357 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.371+0000 7f28367fc640 1 -- 172.21.15.176:0/2513601159 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1614001607 0 0) 0x7f28380b0910 con 0x7f283813fe30 2024-03-29T11:20:50.358 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.371+0000 7f28367fc640 10 monclient(hunting): _negotiate 2024-03-29T11:20:50.358 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.371+0000 7f28367fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:50.358 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.371+0000 7f28367fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:50.358 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.371+0000 7f28367fc640 10 monclient(hunting): my global_id is 4109 2024-03-29T11:20:50.358 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.371+0000 7f28367fc640 1 -- 172.21.15.176:0/2513601159 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28381403a0 con 0x7f283813fe30 2024-03-29T11:20:50.358 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.371+0000 7f28367fc640 1 -- 172.21.15.176:0/2513601159 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 618159563 0 0) 0x7f2820003520 con 0x7f2838066b90 2024-03-29T11:20:50.358 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.371+0000 7f28367fc640 1 -- 172.21.15.176:0/2513601159 >> v1:172.21.15.114:6790/0 conn(0x7f283813fe30 legacy=0x7f2838070da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:50.358 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.371+0000 7f28367fc640 1 -- 172.21.15.176:0/2513601159 >> v1:172.21.15.114:6789/0 conn(0x7f283813d0f0 legacy=0x7f283814b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:50.358 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.371+0000 7f28367fc640 10 monclient: _finish_hunting 0 2024-03-29T11:20:50.358 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.371+0000 7f28367fc640 1 monclient: found mon.noname-c 2024-03-29T11:20:50.358 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.371+0000 7f28367fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:50.358 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.371+0000 7f28367fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.176:6789/0 2024-03-29T11:20:50.358 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.371+0000 7f28367fc640 1 -- 172.21.15.176:0/2513601159 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28380b0bf0 con 0x7f2838066b90 2024-03-29T11:20:50.358 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.371+0000 7f28367fc640 10 monclient: _finish_auth 0 2024-03-29T11:20:50.359 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.371+0000 7f28367fc640 10 monclient: _check_auth_tickets 2024-03-29T11:20:50.359 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.371+0000 7f28367fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:50.359 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.371+0000 7f283dbd6640 5 monclient: authenticate success, global_id 4114 2024-03-29T11:20:50.359 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.371+0000 7f283dbd6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:50.359 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.371+0000 7f28367fc640 1 -- 172.21.15.176:0/2513601159 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f2828001690 con 0x7f2838066b90 2024-03-29T11:20:50.359 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.371+0000 7f28367fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:50.359 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.371+0000 7f28367fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:20:50.359 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.371+0000 7f28367fc640 10 monclient: dump: 2024-03-29T11:20:50.359 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-03-29T11:20:50.359 INFO:teuthology.orchestra.run.smithi176.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:20:50.359 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:50.359 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:50.359 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:50.359 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T11:20:50.359 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:20:50.359 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:20:50.360 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:20:50.360 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T11:20:50.360 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.371+0000 7f283dbd6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:50.360 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.371+0000 7f28367fc640 1 -- 172.21.15.176:0/2513601159 <== mon.1 v1:172.21.15.176:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2828001870 con 0x7f2838066b90 2024-03-29T11:20:50.360 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.371+0000 7f28367fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:50.360 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.371+0000 7f28367fc640 1 -- 172.21.15.176:0/2513601159 <== mon.1 v1:172.21.15.176:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f2828001c00 con 0x7f2838066b90 2024-03-29T11:20:50.360 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.371+0000 7f283dbd6640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:50.360 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.371+0000 7f283dbd6640 4 set_mon_vals no callback set 2024-03-29T11:20:50.361 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.371+0000 7f28367fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:50.361 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.371+0000 7f28367fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:20:50.361 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.371+0000 7f28367fc640 10 monclient: dump: 2024-03-29T11:20:50.361 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-03-29T11:20:50.361 INFO:teuthology.orchestra.run.smithi176.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:20:50.361 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:50.361 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:50.361 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:50.361 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T11:20:50.361 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:20:50.361 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:20:50.361 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:20:50.361 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T11:20:50.362 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.371+0000 7f283dbd6640 10 monclient: shutdown 2024-03-29T11:20:50.362 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.371+0000 7f283dbd6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:50.362 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.371+0000 7f283dbd6640 1 -- 172.21.15.176:0/2513601159 >> v1:172.21.15.176:6789/0 conn(0x7f2838066b90 legacy=0x7f2838140830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:50.362 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.371+0000 7f283dbd6640 1 -- 172.21.15.176:0/2513601159 shutdown_connections 2024-03-29T11:20:50.362 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.371+0000 7f283dbd6640 1 -- 172.21.15.176:0/2513601159 >> 172.21.15.176:0/2513601159 conn(0x7f28380b4970 msgr2=0x7f28380b4d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:50.362 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.371+0000 7f283dbd6640 1 -- 172.21.15.176:0/2513601159 shutdown_connections 2024-03-29T11:20:50.362 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.371+0000 7f283dbd6640 1 -- 172.21.15.176:0/2513601159 wait complete. 2024-03-29T11:20:50.362 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.372+0000 7f283dbd6640 10 monclient: build_initial_monmap 2024-03-29T11:20:50.362 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.372+0000 7f283dbd6640 10 monclient: monmap: 2024-03-29T11:20:50.362 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 0 2024-03-29T11:20:50.362 INFO:teuthology.orchestra.run.smithi176.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:50.363 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 0.000000 2024-03-29T11:20:50.363 INFO:teuthology.orchestra.run.smithi176.stderr:created 0.000000 2024-03-29T11:20:50.363 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:50.363 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T11:20:50.363 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:20:50.363 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T11:20:50.363 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T11:20:50.363 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T11:20:50.363 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.372+0000 7f283dbd6640 1 librados: starting msgr at 2024-03-29T11:20:50.364 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.372+0000 7f283dbd6640 1 librados: starting objecter 2024-03-29T11:20:50.364 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.372+0000 7f283dbd6640 1 Processor -- start 2024-03-29T11:20:50.364 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.373+0000 7f283dbd6640 1 -- start start 2024-03-29T11:20:50.364 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.373+0000 7f283dbd6640 1 librados: setting wanted keys 2024-03-29T11:20:50.364 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.373+0000 7f283dbd6640 1 librados: calling monclient init 2024-03-29T11:20:50.364 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.373+0000 7f283dbd6640 10 monclient: init 2024-03-29T11:20:50.364 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.373+0000 7f283dbd6640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:50.364 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.373+0000 7f283dbd6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:20:50.364 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.373+0000 7f283dbd6640 10 monclient(hunting): picked mon.noname-a con 0x7f2838066b90 addr v1:172.21.15.114:6789/0 2024-03-29T11:20:50.364 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.373+0000 7f283dbd6640 10 monclient(hunting): picked mon.noname-b con 0x7f283813d0f0 addr v1:172.21.15.176:6789/0 2024-03-29T11:20:50.364 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.373+0000 7f283dbd6640 10 monclient(hunting): picked mon.noname-c con 0x7f283813fe30 addr v1:172.21.15.114:6790/0 2024-03-29T11:20:50.365 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.373+0000 7f283dbd6640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2838070760 con 0x7f2838066b90 2024-03-29T11:20:50.365 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.373+0000 7f283dbd6640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2838142d20 con 0x7f283813d0f0 2024-03-29T11:20:50.365 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.373+0000 7f283dbd6640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2838070310 con 0x7f283813fe30 2024-03-29T11:20:50.365 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.373+0000 7f283dbd6640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:50.365 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.373+0000 7f283dbd6640 10 monclient(hunting): authenticate will time out at 915.714843s 2024-03-29T11:20:50.365 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.373+0000 7f2836ffd640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f283813d0f0 0x7f2838143500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.176:49030/0 (socket says 172.21.15.176:49030) 2024-03-29T11:20:50.365 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.373+0000 7f28377fe640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f2838066b90 0x7f2838135e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.176:51596/0 (socket says 172.21.15.176:51596) 2024-03-29T11:20:50.365 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.373+0000 7f2836ffd640 1 -- 172.21.15.176:0/701726126 learned_addr learned my addr 172.21.15.176:0/701726126 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-29T11:20:50.365 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.373+0000 7f281ffff640 10 client.?.objecter ms_handle_connect 0x7f283813d0f0 2024-03-29T11:20:50.366 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.373+0000 7f281ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:50.366 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.373+0000 7f281ffff640 10 client.?.objecter ms_handle_connect 0x7f2838066b90 2024-03-29T11:20:50.366 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.373+0000 7f281ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:50.366 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.373+0000 7f281ffff640 10 client.?.objecter ms_handle_connect 0x7f283813fe30 2024-03-29T11:20:50.366 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.373+0000 7f281ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:50.366 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.373+0000 7f281ffff640 1 -- 172.21.15.176:0/701726126 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2804800050 0 0) 0x7f2838142d20 con 0x7f283813d0f0 2024-03-29T11:20:50.366 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.373+0000 7f281ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:20:50.366 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.373+0000 7f281ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:50.366 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.373+0000 7f281ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:50.366 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.373+0000 7f281ffff640 10 monclient(hunting): my global_id is 4117 2024-03-29T11:20:50.366 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.374+0000 7f281ffff640 1 -- 172.21.15.176:0/701726126 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2824004eb0 con 0x7f283813d0f0 2024-03-29T11:20:50.366 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.374+0000 7f281ffff640 1 -- 172.21.15.176:0/701726126 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2448447750 0 0) 0x7f2838070760 con 0x7f2838066b90 2024-03-29T11:20:50.367 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.374+0000 7f281ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:20:50.367 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.374+0000 7f281ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:50.367 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.374+0000 7f281ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:50.367 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.374+0000 7f281ffff640 10 monclient(hunting): my global_id is 4122 2024-03-29T11:20:50.367 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.374+0000 7f281ffff640 1 -- 172.21.15.176:0/701726126 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2838142d20 con 0x7f2838066b90 2024-03-29T11:20:50.367 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.374+0000 7f281ffff640 1 -- 172.21.15.176:0/701726126 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 193525678 0 0) 0x7f2838070310 con 0x7f283813fe30 2024-03-29T11:20:50.367 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.374+0000 7f281ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:20:50.367 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.374+0000 7f281ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:50.367 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.374+0000 7f281ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:50.367 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.374+0000 7f281ffff640 10 monclient(hunting): my global_id is 4112 2024-03-29T11:20:50.368 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.374+0000 7f281ffff640 1 -- 172.21.15.176:0/701726126 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2838070760 con 0x7f283813fe30 2024-03-29T11:20:50.368 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.374+0000 7f281ffff640 1 -- 172.21.15.176:0/701726126 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 30381955 0 0) 0x7f2824004eb0 con 0x7f283813d0f0 2024-03-29T11:20:50.368 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.374+0000 7f281ffff640 1 -- 172.21.15.176:0/701726126 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2838070310 con 0x7f283813d0f0 2024-03-29T11:20:50.368 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.374+0000 7f281ffff640 1 -- 172.21.15.176:0/701726126 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f282c001c90 con 0x7f283813d0f0 2024-03-29T11:20:50.368 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.374+0000 7f281ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:50.368 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.374+0000 7f281ffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:20:50.368 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.374+0000 7f281ffff640 10 monclient(hunting): dump: 2024-03-29T11:20:50.368 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-03-29T11:20:50.368 INFO:teuthology.orchestra.run.smithi176.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:20:50.368 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:50.368 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:50.368 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:50.368 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T11:20:50.368 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:20:50.369 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:20:50.369 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:20:50.369 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T11:20:50.369 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.374+0000 7f281ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:50.369 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.374+0000 7f281ffff640 1 -- 172.21.15.176:0/701726126 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1062970772 0 0) 0x7f2838142d20 con 0x7f2838066b90 2024-03-29T11:20:50.369 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.374+0000 7f281ffff640 1 -- 172.21.15.176:0/701726126 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2824004eb0 con 0x7f2838066b90 2024-03-29T11:20:50.369 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.374+0000 7f281ffff640 1 -- 172.21.15.176:0/701726126 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f2828001b80 con 0x7f2838066b90 2024-03-29T11:20:50.369 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.374+0000 7f281ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:50.369 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.374+0000 7f281ffff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:20:50.369 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.374+0000 7f281ffff640 10 monclient(hunting): dump: 2024-03-29T11:20:50.370 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-03-29T11:20:50.370 INFO:teuthology.orchestra.run.smithi176.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:20:50.370 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:50.370 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:50.370 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:50.370 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T11:20:50.370 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:20:50.370 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:20:50.370 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:20:50.370 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T11:20:50.370 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.374+0000 7f281ffff640 1 -- 172.21.15.176:0/701726126 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 55156792 0 0) 0x7f2838070760 con 0x7f283813fe30 2024-03-29T11:20:50.371 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.374+0000 7f281ffff640 1 -- 172.21.15.176:0/701726126 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2838142d20 con 0x7f283813fe30 2024-03-29T11:20:50.371 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.374+0000 7f281ffff640 1 -- 172.21.15.176:0/701726126 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f28300016d0 con 0x7f283813fe30 2024-03-29T11:20:50.371 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.374+0000 7f281ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:50.371 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.374+0000 7f281ffff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:20:50.371 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.374+0000 7f281ffff640 10 monclient(hunting): dump: 2024-03-29T11:20:50.371 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-03-29T11:20:50.371 INFO:teuthology.orchestra.run.smithi176.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:20:50.371 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:50.371 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:50.371 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:50.371 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T11:20:50.371 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:20:50.372 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:20:50.372 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:20:50.372 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T11:20:50.372 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.374+0000 7f281ffff640 1 -- 172.21.15.176:0/701726126 <== mon.1 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1270404856 0 0) 0x7f2838070310 con 0x7f283813d0f0 2024-03-29T11:20:50.372 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.374+0000 7f281ffff640 1 -- 172.21.15.176:0/701726126 >> v1:172.21.15.114:6790/0 conn(0x7f283813fe30 legacy=0x7f28381e27e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:50.372 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.374+0000 7f281ffff640 1 -- 172.21.15.176:0/701726126 >> v1:172.21.15.114:6789/0 conn(0x7f2838066b90 legacy=0x7f2838135e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:50.372 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.374+0000 7f281ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:20:50.372 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.374+0000 7f281ffff640 1 monclient: found mon.b 2024-03-29T11:20:50.372 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.374+0000 7f281ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:50.372 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.374+0000 7f281ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:20:50.372 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.374+0000 7f281ffff640 1 -- 172.21.15.176:0/701726126 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28380704a0 con 0x7f283813d0f0 2024-03-29T11:20:50.372 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.374+0000 7f281ffff640 10 monclient: _finish_auth 0 2024-03-29T11:20:50.372 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.374+0000 7f281ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:20:50.372 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.374+0000 7f281ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:50.373 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.374+0000 7f283dbd6640 5 monclient: authenticate success, global_id 4117 2024-03-29T11:20:50.373 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.374+0000 7f283dbd6640 10 monclient: _renew_subs 2024-03-29T11:20:50.373 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.374+0000 7f283dbd6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:20:50.373 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.374+0000 7f283dbd6640 1 -- 172.21.15.176:0/701726126 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2838136570 con 0x7f283813d0f0 2024-03-29T11:20:50.373 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.374+0000 7f283dbd6640 10 client.4117.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:50.373 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.374+0000 7f283dbd6640 10 monclient: _renew_subs 2024-03-29T11:20:50.373 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.374+0000 7f283dbd6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:20:50.373 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.374+0000 7f283dbd6640 1 -- 172.21.15.176:0/701726126 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2838141a60 con 0x7f283813d0f0 2024-03-29T11:20:50.373 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.374+0000 7f283dbd6640 1 librados: init done 2024-03-29T11:20:50.373 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.374+0000 7f281ffff640 1 -- 172.21.15.176:0/701726126 <== mon.1 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f282c003990 con 0x7f283813d0f0 2024-03-29T11:20:50.373 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.374+0000 7f281ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:50.374 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.374+0000 7f281ffff640 1 -- 172.21.15.176:0/701726126 <== mon.1 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f282c003ce0 con 0x7f283813d0f0 2024-03-29T11:20:50.374 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.374+0000 7f281ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:50.374 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.374+0000 7f281ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:20:50.374 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.374+0000 7f281ffff640 10 monclient: dump: 2024-03-29T11:20:50.374 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-03-29T11:20:50.374 INFO:teuthology.orchestra.run.smithi176.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:20:50.374 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:50.374 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:50.374 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:50.374 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T11:20:50.374 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:20:50.374 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:20:50.374 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:20:50.374 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T11:20:50.375 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.374+0000 7f2835ffb640 4 set_mon_vals no callback set 2024-03-29T11:20:50.375 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.374+0000 7f281ffff640 1 -- 172.21.15.176:0/701726126 <== mon.1 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f282c004290 con 0x7f283813d0f0 2024-03-29T11:20:50.375 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.374+0000 7f281ffff640 1 -- 172.21.15.176:0/701726126 <== mon.1 v1:172.21.15.176:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (unknown 1646432991 0 0) 0x7f282c006990 con 0x7f283813d0f0 2024-03-29T11:20:50.375 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.374+0000 7f281ffff640 10 client.4117.objecter ms_dispatch 0x7f2838000b90 osd_map(3..3 src has 1..3) v4 2024-03-29T11:20:50.375 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.374+0000 7f281ffff640 3 client.4117.objecter handle_osd_map got epochs [3,3] > 0 2024-03-29T11:20:50.375 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.374+0000 7f281ffff640 3 client.4117.objecter handle_osd_map decoding full epoch 3 2024-03-29T11:20:50.375 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.374+0000 7f281ffff640 20 client.4117.objecter dump_active .. 0 homeless 2024-03-29T11:20:50.375 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.375+0000 7f283dbd6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:50.375 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.375+0000 7f283dbd6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:50.375 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.375+0000 7f283dbd6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:20:50.375 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.375+0000 7f283dbd6640 1 -- 172.21.15.176:0/701726126 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2818005080 con 0x7f283813d0f0 2024-03-29T11:20:50.375 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.377+0000 7f281ffff640 1 -- 172.21.15.176:0/701726126 <== mon.1 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f282c004770 con 0x7f283813d0f0 2024-03-29T11:20:50.376 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.377+0000 7f281ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:50.376 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.377+0000 7f281ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:50.426 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.449+0000 7f283dbd6640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "5882dadb-e7a8-4604-ba9a-418013c106c8", "id": 1}] 2024-03-29T11:20:50.426 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.449+0000 7f283dbd6640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "5882dadb-e7a8-4604-ba9a-418013c106c8", "id": 1}] 2024-03-29T11:20:50.426 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.450+0000 7f283dbd6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:20:50.426 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.450+0000 7f283dbd6640 1 -- 172.21.15.176:0/701726126 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd new", "uuid": "5882dadb-e7a8-4604-ba9a-418013c106c8", "id": 1} v 0) v1 -- 0x7f28180058b0 con 0x7f283813d0f0 2024-03-29T11:20:50.430 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.454+0000 7f281ffff640 1 -- 172.21.15.176:0/701726126 <== mon.1 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "5882dadb-e7a8-4604-ba9a-418013c106c8", "id": 1}]=0 v4) v1 ==== 112+0+2 (unknown 3643754072 0 940483276) 0x7f282c005b40 con 0x7f283813d0f0 2024-03-29T11:20:50.430 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.454+0000 7f281ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "5882dadb-e7a8-4604-ba9a-418013c106c8", "id": 1}] 2024-03-29T11:20:50.430 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.454+0000 7f281ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:50.431 INFO:teuthology.orchestra.run.smithi176.stdout:1 2024-03-29T11:20:50.432 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.454+0000 7f283dbd6640 10 librados: watch_flush enter 2024-03-29T11:20:50.432 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.455+0000 7f283dbd6640 10 librados: watch_flush exit 2024-03-29T11:20:50.432 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.455+0000 7f283dbd6640 20 client.4117.objecter shutdown clearing up homeless session... 2024-03-29T11:20:50.432 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.455+0000 7f283dbd6640 10 client.4117.objecter successfully canceled tick 2024-03-29T11:20:50.432 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.455+0000 7f283dbd6640 10 monclient: shutdown 2024-03-29T11:20:50.432 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.455+0000 7f283dbd6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:50.432 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.455+0000 7f283dbd6640 1 -- 172.21.15.176:0/701726126 >> v1:172.21.15.176:6789/0 conn(0x7f283813d0f0 legacy=0x7f2838143500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:50.432 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.455+0000 7f283dbd6640 1 -- 172.21.15.176:0/701726126 shutdown_connections 2024-03-29T11:20:50.432 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.455+0000 7f283dbd6640 1 -- 172.21.15.176:0/701726126 >> 172.21.15.176:0/701726126 conn(0x7f28380b4970 msgr2=0x7f283813a3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:50.432 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.455+0000 7f283dbd6640 1 -- 172.21.15.176:0/701726126 shutdown_connections 2024-03-29T11:20:50.433 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.455+0000 7f283dbd6640 1 -- 172.21.15.176:0/701726126 wait complete. 2024-03-29T11:20:50.433 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.455+0000 7f283dbd6640 1 librados: shutdown 2024-03-29T11:20:50.442 DEBUG:teuthology.orchestra.run.smithi176:> sudo ceph --cluster ceph osd new d7e8c409-bc66-4e4a-bbdb-420bad31388e 2 2024-03-29T11:20:50.502 INFO:tasks.ceph.mgr.y.smithi114.stderr:2024-03-29T11:20:50.526+0000 7ffbece52040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T11:20:50.539 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-03-29T11:20:50.562+0000 7fedaa420040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T11:20:50.542 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.566+0000 7f9fe8814640 10 monclient: get_monmap_and_config 2024-03-29T11:20:50.542 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.566+0000 7f9fe8814640 10 monclient: build_initial_monmap 2024-03-29T11:20:50.542 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.566+0000 7f9fe8814640 10 monclient: monmap: 2024-03-29T11:20:50.542 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 0 2024-03-29T11:20:50.543 INFO:teuthology.orchestra.run.smithi176.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:20:50.543 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-29T11:20:50.567181+0000 2024-03-29T11:20:50.543 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-29T11:20:50.567181+0000 2024-03-29T11:20:50.543 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:50.543 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T11:20:50.543 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:20:50.543 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T11:20:50.543 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T11:20:50.543 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T11:20:50.553 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.566+0000 7f9fe8814640 1 Processor -- start 2024-03-29T11:20:50.553 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.566+0000 7f9fe8814640 1 -- start start 2024-03-29T11:20:50.553 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.566+0000 7f9fe8814640 10 monclient: init 2024-03-29T11:20:50.553 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.567+0000 7f9fe8814640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:50.553 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.567+0000 7f9fe8814640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:20:50.553 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.567+0000 7f9fe8814640 10 monclient(hunting): picked mon.noname-a con 0x7f9fe409ac40 addr v1:172.21.15.114:6789/0 2024-03-29T11:20:50.553 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.567+0000 7f9fe8814640 10 monclient(hunting): picked mon.noname-b con 0x7f9fe409c870 addr v1:172.21.15.114:6790/0 2024-03-29T11:20:50.553 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.567+0000 7f9fe8814640 10 monclient(hunting): picked mon.noname-c con 0x7f9fe40b4360 addr v1:172.21.15.176:6789/0 2024-03-29T11:20:50.553 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.567+0000 7f9fe8814640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fe409dec0 con 0x7f9fe409ac40 2024-03-29T11:20:50.553 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.567+0000 7f9fe8814640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fe409da30 con 0x7f9fe40b4360 2024-03-29T11:20:50.553 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.567+0000 7f9fe8814640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fe40b39c0 con 0x7f9fe409c870 2024-03-29T11:20:50.553 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.567+0000 7f9fe8814640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:50.554 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.567+0000 7f9fe8814640 10 monclient(hunting): authenticate will time out at 915.908663s 2024-03-29T11:20:50.554 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.567+0000 7f9fe2d9d640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f9fe40b4360 0x7f9fe414d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.176:49034/0 (socket says 172.21.15.176:49034) 2024-03-29T11:20:50.554 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.567+0000 7f9fe2d9d640 1 -- 172.21.15.176:0/2486710139 learned_addr learned my addr 172.21.15.176:0/2486710139 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-29T11:20:50.554 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.567+0000 7f9fe1d9b640 1 -- 172.21.15.176:0/2486710139 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2731236656 0 0) 0x7f9fe409da30 con 0x7f9fe40b4360 2024-03-29T11:20:50.554 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.567+0000 7f9fe1d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:20:50.554 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.567+0000 7f9fe1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:50.554 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.567+0000 7f9fe1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:50.554 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.567+0000 7f9fe1d9b640 10 monclient(hunting): my global_id is 4120 2024-03-29T11:20:50.554 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.567+0000 7f9fe1d9b640 1 -- 172.21.15.176:0/2486710139 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fd0003520 con 0x7f9fe40b4360 2024-03-29T11:20:50.554 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.567+0000 7f9fe1d9b640 1 -- 172.21.15.176:0/2486710139 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2664333652 0 0) 0x7f9fd4001110 con 0x7f9fe409ac40 2024-03-29T11:20:50.554 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.567+0000 7f9fe1d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:20:50.554 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.567+0000 7f9fe1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:50.554 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.567+0000 7f9fe1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:50.554 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.567+0000 7f9fe1d9b640 10 monclient(hunting): my global_id is 4125 2024-03-29T11:20:50.555 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.567+0000 7f9fe1d9b640 1 -- 172.21.15.176:0/2486710139 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fe409da30 con 0x7f9fe409ac40 2024-03-29T11:20:50.555 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.567+0000 7f9fe1d9b640 1 -- 172.21.15.176:0/2486710139 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3977425316 0 0) 0x7f9fe40b39c0 con 0x7f9fe409c870 2024-03-29T11:20:50.555 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.568+0000 7f9fe1d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:20:50.555 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.568+0000 7f9fe1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:50.555 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.568+0000 7f9fe1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:50.555 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.568+0000 7f9fe1d9b640 10 monclient(hunting): my global_id is 4115 2024-03-29T11:20:50.555 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.568+0000 7f9fe1d9b640 1 -- 172.21.15.176:0/2486710139 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fd4001110 con 0x7f9fe409c870 2024-03-29T11:20:50.555 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.568+0000 7f9fe1d9b640 1 -- 172.21.15.176:0/2486710139 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1354529134 0 0) 0x7f9fd0003520 con 0x7f9fe40b4360 2024-03-29T11:20:50.555 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.568+0000 7f9fe1d9b640 1 -- 172.21.15.176:0/2486710139 >> v1:172.21.15.114:6790/0 conn(0x7f9fe409c870 legacy=0x7f9fe413f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:50.555 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.568+0000 7f9fe1d9b640 1 -- 172.21.15.176:0/2486710139 >> v1:172.21.15.114:6789/0 conn(0x7f9fe409ac40 legacy=0x7f9fe40b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:50.555 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.568+0000 7f9fe1d9b640 10 monclient: _finish_hunting 0 2024-03-29T11:20:50.555 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.568+0000 7f9fe1d9b640 1 monclient: found mon.noname-c 2024-03-29T11:20:50.556 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.568+0000 7f9fe1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:50.556 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.568+0000 7f9fe1d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.176:6789/0 2024-03-29T11:20:50.556 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.568+0000 7f9fe1d9b640 1 -- 172.21.15.176:0/2486710139 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fe40b5160 con 0x7f9fe40b4360 2024-03-29T11:20:50.556 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.568+0000 7f9fe1d9b640 10 monclient: _finish_auth 0 2024-03-29T11:20:50.556 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.568+0000 7f9fe1d9b640 10 monclient: _check_auth_tickets 2024-03-29T11:20:50.556 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.568+0000 7f9fe1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:50.556 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.568+0000 7f9fe8814640 5 monclient: authenticate success, global_id 4120 2024-03-29T11:20:50.556 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.568+0000 7f9fe1d9b640 1 -- 172.21.15.176:0/2486710139 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f9fdc001a10 con 0x7f9fe40b4360 2024-03-29T11:20:50.556 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.568+0000 7f9fe8814640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:50.556 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.568+0000 7f9fe1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:50.556 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.568+0000 7f9fe1d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:20:50.556 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.568+0000 7f9fe1d9b640 10 monclient: dump: 2024-03-29T11:20:50.557 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-03-29T11:20:50.557 INFO:teuthology.orchestra.run.smithi176.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:20:50.557 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:50.557 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:50.557 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:50.557 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T11:20:50.557 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:20:50.557 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:20:50.557 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:20:50.557 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T11:20:50.557 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.568+0000 7f9fe8814640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:50.557 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.568+0000 7f9fe1d9b640 1 -- 172.21.15.176:0/2486710139 <== mon.1 v1:172.21.15.176:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9fdc001bf0 con 0x7f9fe40b4360 2024-03-29T11:20:50.558 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.568+0000 7f9fe1d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:50.558 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.568+0000 7f9fe8814640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:50.558 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.568+0000 7f9fe1d9b640 1 -- 172.21.15.176:0/2486710139 <== mon.1 v1:172.21.15.176:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f9fdc002000 con 0x7f9fe40b4360 2024-03-29T11:20:50.558 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.568+0000 7f9fe1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:50.558 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.568+0000 7f9fe8814640 4 set_mon_vals no callback set 2024-03-29T11:20:50.558 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.568+0000 7f9fe1d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:20:50.558 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.568+0000 7f9fe1d9b640 10 monclient: dump: 2024-03-29T11:20:50.558 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-03-29T11:20:50.558 INFO:teuthology.orchestra.run.smithi176.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:20:50.558 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:50.558 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:50.558 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:50.558 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T11:20:50.559 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:20:50.559 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:20:50.559 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:20:50.559 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T11:20:50.559 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.568+0000 7f9fe8814640 10 monclient: shutdown 2024-03-29T11:20:50.559 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.568+0000 7f9fe8814640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:50.559 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.568+0000 7f9fe8814640 1 -- 172.21.15.176:0/2486710139 >> v1:172.21.15.176:6789/0 conn(0x7f9fe40b4360 legacy=0x7f9fe414d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:50.559 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.568+0000 7f9fe8814640 1 -- 172.21.15.176:0/2486710139 shutdown_connections 2024-03-29T11:20:50.559 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.568+0000 7f9fe8814640 1 -- 172.21.15.176:0/2486710139 >> 172.21.15.176:0/2486710139 conn(0x7f9fe4066b00 msgr2=0x7f9fe4066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:50.559 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.568+0000 7f9fe8814640 1 -- 172.21.15.176:0/2486710139 shutdown_connections 2024-03-29T11:20:50.559 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.568+0000 7f9fe8814640 1 -- 172.21.15.176:0/2486710139 wait complete. 2024-03-29T11:20:50.559 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.569+0000 7f9fe8814640 10 monclient: build_initial_monmap 2024-03-29T11:20:50.559 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.569+0000 7f9fe8814640 10 monclient: monmap: 2024-03-29T11:20:50.560 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 0 2024-03-29T11:20:50.560 INFO:teuthology.orchestra.run.smithi176.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:50.560 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 0.000000 2024-03-29T11:20:50.560 INFO:teuthology.orchestra.run.smithi176.stderr:created 0.000000 2024-03-29T11:20:50.560 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:50.560 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T11:20:50.560 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:20:50.560 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T11:20:50.560 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T11:20:50.560 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T11:20:50.560 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.569+0000 7f9fe8814640 1 librados: starting msgr at 2024-03-29T11:20:50.560 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.569+0000 7f9fe8814640 1 librados: starting objecter 2024-03-29T11:20:50.560 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.569+0000 7f9fe8814640 1 Processor -- start 2024-03-29T11:20:50.561 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.570+0000 7f9fe8814640 1 -- start start 2024-03-29T11:20:50.561 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.570+0000 7f9fe8814640 1 librados: setting wanted keys 2024-03-29T11:20:50.561 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.570+0000 7f9fe8814640 1 librados: calling monclient init 2024-03-29T11:20:50.561 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.570+0000 7f9fe8814640 10 monclient: init 2024-03-29T11:20:50.561 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.570+0000 7f9fe8814640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:50.561 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.570+0000 7f9fe8814640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:20:50.561 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.570+0000 7f9fe8814640 10 monclient(hunting): picked mon.noname-b con 0x7f9fe409ac40 addr v1:172.21.15.176:6789/0 2024-03-29T11:20:50.561 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.570+0000 7f9fe8814640 10 monclient(hunting): picked mon.noname-c con 0x7f9fe4142c20 addr v1:172.21.15.114:6790/0 2024-03-29T11:20:50.561 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.570+0000 7f9fe8814640 10 monclient(hunting): picked mon.noname-a con 0x7f9fe41d7dc0 addr v1:172.21.15.114:6789/0 2024-03-29T11:20:50.561 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.570+0000 7f9fe8814640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fe41d4d30 con 0x7f9fe41d7dc0 2024-03-29T11:20:50.561 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.570+0000 7f9fe8814640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fe41430b0 con 0x7f9fe409ac40 2024-03-29T11:20:50.561 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.570+0000 7f9fe8814640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fe41d71b0 con 0x7f9fe4142c20 2024-03-29T11:20:50.562 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.570+0000 7f9fe8814640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:50.562 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.570+0000 7f9fe8814640 10 monclient(hunting): authenticate will time out at 915.911707s 2024-03-29T11:20:50.562 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.570+0000 7f9fe259c640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f9fe409ac40 0x7f9fe4143710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.176:49048/0 (socket says 172.21.15.176:49048) 2024-03-29T11:20:50.562 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.570+0000 7f9fe259c640 1 -- 172.21.15.176:0/3259740371 learned_addr learned my addr 172.21.15.176:0/3259740371 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-29T11:20:50.562 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.570+0000 7f9fdaffd640 10 client.?.objecter ms_handle_connect 0x7f9fe409ac40 2024-03-29T11:20:50.562 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.570+0000 7f9fdaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:50.562 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.570+0000 7f9fdaffd640 10 client.?.objecter ms_handle_connect 0x7f9fe41d7dc0 2024-03-29T11:20:50.562 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.570+0000 7f9fdaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:50.562 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.570+0000 7f9fdaffd640 10 client.?.objecter ms_handle_connect 0x7f9fe4142c20 2024-03-29T11:20:50.562 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.570+0000 7f9fdaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:50.562 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.570+0000 7f9fdaffd640 1 -- 172.21.15.176:0/3259740371 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4063318437 0 0) 0x7f9fe41430b0 con 0x7f9fe409ac40 2024-03-29T11:20:50.562 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.570+0000 7f9fdaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:20:50.563 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.570+0000 7f9fdaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:50.563 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.570+0000 7f9fdaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:50.563 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.570+0000 7f9fdaffd640 10 monclient(hunting): my global_id is 4123 2024-03-29T11:20:50.563 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.570+0000 7f9fdaffd640 1 -- 172.21.15.176:0/3259740371 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fc4004eb0 con 0x7f9fe409ac40 2024-03-29T11:20:50.563 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.570+0000 7f9fdaffd640 1 -- 172.21.15.176:0/3259740371 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1676023074 0 0) 0x7f9fe41d4d30 con 0x7f9fe41d7dc0 2024-03-29T11:20:50.563 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.570+0000 7f9fdaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:20:50.563 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.570+0000 7f9fdaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:50.563 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.570+0000 7f9fdaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:50.563 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.570+0000 7f9fdaffd640 10 monclient(hunting): my global_id is 4128 2024-03-29T11:20:50.563 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.570+0000 7f9fdaffd640 1 -- 172.21.15.176:0/3259740371 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fe41430b0 con 0x7f9fe41d7dc0 2024-03-29T11:20:50.563 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.570+0000 7f9fdaffd640 1 -- 172.21.15.176:0/3259740371 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1997764531 0 0) 0x7f9fe41d71b0 con 0x7f9fe4142c20 2024-03-29T11:20:50.563 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.570+0000 7f9fdaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:20:50.563 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.570+0000 7f9fdaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:50.563 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.570+0000 7f9fdaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:50.564 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.570+0000 7f9fdaffd640 10 monclient(hunting): my global_id is 4118 2024-03-29T11:20:50.564 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.570+0000 7f9fdaffd640 1 -- 172.21.15.176:0/3259740371 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fe41d4d30 con 0x7f9fe4142c20 2024-03-29T11:20:50.564 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.571+0000 7f9fdaffd640 1 -- 172.21.15.176:0/3259740371 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2297540967 0 0) 0x7f9fc4004eb0 con 0x7f9fe409ac40 2024-03-29T11:20:50.564 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.571+0000 7f9fdaffd640 1 -- 172.21.15.176:0/3259740371 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9fe41d71b0 con 0x7f9fe409ac40 2024-03-29T11:20:50.564 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.571+0000 7f9fdaffd640 1 -- 172.21.15.176:0/3259740371 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f9fd40019a0 con 0x7f9fe409ac40 2024-03-29T11:20:50.564 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.571+0000 7f9fdaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:50.564 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.571+0000 7f9fdaffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:20:50.564 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.571+0000 7f9fdaffd640 10 monclient(hunting): dump: 2024-03-29T11:20:50.564 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-03-29T11:20:50.564 INFO:teuthology.orchestra.run.smithi176.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:20:50.564 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:50.564 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:50.564 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:50.564 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T11:20:50.565 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:20:50.565 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:20:50.565 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:20:50.565 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T11:20:50.565 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.571+0000 7f9fdaffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:50.565 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.571+0000 7f9fdaffd640 1 -- 172.21.15.176:0/3259740371 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3119032663 0 0) 0x7f9fe41430b0 con 0x7f9fe41d7dc0 2024-03-29T11:20:50.565 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.571+0000 7f9fdaffd640 1 -- 172.21.15.176:0/3259740371 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9fc4004eb0 con 0x7f9fe41d7dc0 2024-03-29T11:20:50.565 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.571+0000 7f9fdaffd640 1 -- 172.21.15.176:0/3259740371 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f9fdc001f00 con 0x7f9fe41d7dc0 2024-03-29T11:20:50.565 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.571+0000 7f9fdaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:50.565 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.571+0000 7f9fdaffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:20:50.565 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.571+0000 7f9fdaffd640 10 monclient(hunting): dump: 2024-03-29T11:20:50.565 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-03-29T11:20:50.565 INFO:teuthology.orchestra.run.smithi176.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:20:50.565 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:50.566 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:50.566 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:50.566 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T11:20:50.566 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:20:50.566 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:20:50.566 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:20:50.566 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T11:20:50.566 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.571+0000 7f9fdaffd640 1 -- 172.21.15.176:0/3259740371 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3150456785 0 0) 0x7f9fe41d4d30 con 0x7f9fe4142c20 2024-03-29T11:20:50.566 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.571+0000 7f9fdaffd640 1 -- 172.21.15.176:0/3259740371 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9fe41430b0 con 0x7f9fe4142c20 2024-03-29T11:20:50.566 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.571+0000 7f9fdaffd640 1 -- 172.21.15.176:0/3259740371 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f9fcc001830 con 0x7f9fe4142c20 2024-03-29T11:20:50.566 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.571+0000 7f9fdaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:50.566 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.571+0000 7f9fdaffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:20:50.566 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.571+0000 7f9fdaffd640 10 monclient(hunting): dump: 2024-03-29T11:20:50.566 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-03-29T11:20:50.567 INFO:teuthology.orchestra.run.smithi176.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:20:50.567 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:50.567 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:50.567 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:50.567 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T11:20:50.567 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:20:50.567 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:20:50.567 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:20:50.567 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T11:20:50.567 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.571+0000 7f9fdaffd640 1 -- 172.21.15.176:0/3259740371 <== mon.1 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1701592384 0 0) 0x7f9fe41d71b0 con 0x7f9fe409ac40 2024-03-29T11:20:50.567 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.571+0000 7f9fdaffd640 1 -- 172.21.15.176:0/3259740371 >> v1:172.21.15.114:6790/0 conn(0x7f9fe4142c20 legacy=0x7f9fe41d76b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:50.567 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.571+0000 7f9fdaffd640 1 -- 172.21.15.176:0/3259740371 >> v1:172.21.15.114:6789/0 conn(0x7f9fe41d7dc0 legacy=0x7f9fe4145a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:50.567 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.571+0000 7f9fdaffd640 10 monclient: _finish_hunting 0 2024-03-29T11:20:50.567 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.571+0000 7f9fdaffd640 1 monclient: found mon.b 2024-03-29T11:20:50.568 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.571+0000 7f9fdaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:50.568 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.571+0000 7f9fdaffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:20:50.568 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.571+0000 7f9fdaffd640 1 -- 172.21.15.176:0/3259740371 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fe4144ce0 con 0x7f9fe409ac40 2024-03-29T11:20:50.568 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.571+0000 7f9fdaffd640 10 monclient: _finish_auth 0 2024-03-29T11:20:50.568 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.571+0000 7f9fdaffd640 10 monclient: _check_auth_tickets 2024-03-29T11:20:50.568 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.571+0000 7f9fdaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:50.568 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.571+0000 7f9fe8814640 5 monclient: authenticate success, global_id 4123 2024-03-29T11:20:50.568 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.571+0000 7f9fe8814640 10 monclient: _renew_subs 2024-03-29T11:20:50.568 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.571+0000 7f9fe8814640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:20:50.568 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.571+0000 7f9fe8814640 1 -- 172.21.15.176:0/3259740371 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fe4145490 con 0x7f9fe409ac40 2024-03-29T11:20:50.568 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.571+0000 7f9fe8814640 10 client.4123.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:50.568 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.571+0000 7f9fe8814640 10 monclient: _renew_subs 2024-03-29T11:20:50.568 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.571+0000 7f9fe8814640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:20:50.568 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.571+0000 7f9fe8814640 1 -- 172.21.15.176:0/3259740371 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fe4146330 con 0x7f9fe409ac40 2024-03-29T11:20:50.568 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.571+0000 7f9fe8814640 1 librados: init done 2024-03-29T11:20:50.569 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.571+0000 7f9fdaffd640 1 -- 172.21.15.176:0/3259740371 <== mon.1 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9fd4003610 con 0x7f9fe409ac40 2024-03-29T11:20:50.569 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.571+0000 7f9fdaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:50.569 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.571+0000 7f9fdaffd640 1 -- 172.21.15.176:0/3259740371 <== mon.1 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f9fd4003920 con 0x7f9fe409ac40 2024-03-29T11:20:50.569 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.571+0000 7f9fdaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:50.569 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.571+0000 7f9fdaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:20:50.569 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.571+0000 7f9fdaffd640 10 monclient: dump: 2024-03-29T11:20:50.569 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-03-29T11:20:50.569 INFO:teuthology.orchestra.run.smithi176.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:20:50.569 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:50.569 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:50.569 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:50.569 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T11:20:50.569 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:20:50.570 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:20:50.570 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:20:50.570 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T11:20:50.570 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.571+0000 7f9fe159a640 4 set_mon_vals no callback set 2024-03-29T11:20:50.570 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.571+0000 7f9fdaffd640 1 -- 172.21.15.176:0/3259740371 <== mon.1 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f9fd4003e10 con 0x7f9fe409ac40 2024-03-29T11:20:50.570 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.571+0000 7f9fdaffd640 1 -- 172.21.15.176:0/3259740371 <== mon.1 v1:172.21.15.176:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (unknown 3388862075 0 0) 0x7f9fd4006590 con 0x7f9fe409ac40 2024-03-29T11:20:50.570 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.571+0000 7f9fdaffd640 10 client.4123.objecter ms_dispatch 0x7f9fe4000b90 osd_map(4..4 src has 1..4) v4 2024-03-29T11:20:50.570 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.571+0000 7f9fdaffd640 3 client.4123.objecter handle_osd_map got epochs [4,4] > 0 2024-03-29T11:20:50.570 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.571+0000 7f9fdaffd640 3 client.4123.objecter handle_osd_map decoding full epoch 4 2024-03-29T11:20:50.570 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.571+0000 7f9fdaffd640 20 client.4123.objecter dump_active .. 0 homeless 2024-03-29T11:20:50.570 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.572+0000 7f9fe8814640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:50.570 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.572+0000 7f9fe8814640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:50.570 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.572+0000 7f9fe8814640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:20:50.570 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.572+0000 7f9fe8814640 1 -- 172.21.15.176:0/3259740371 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fc8005080 con 0x7f9fe409ac40 2024-03-29T11:20:50.571 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.574+0000 7f9fdaffd640 1 -- 172.21.15.176:0/3259740371 <== mon.1 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f9fd4004250 con 0x7f9fe409ac40 2024-03-29T11:20:50.571 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.574+0000 7f9fdaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:50.571 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.574+0000 7f9fdaffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:50.571 INFO:tasks.ceph.mgr.y.smithi114.stderr:2024-03-29T11:20:50.580+0000 7ffbece52040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T11:20:50.595 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-03-29T11:20:50.619+0000 7fedaa420040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T11:20:50.601 INFO:tasks.ceph.mgr.y.smithi114.stderr:2024-03-29T11:20:50.624+0000 7ffbece52040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T11:20:50.623 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.646+0000 7f9fe8814640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "d7e8c409-bc66-4e4a-bbdb-420bad31388e", "id": 2}] 2024-03-29T11:20:50.623 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.646+0000 7f9fe8814640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "d7e8c409-bc66-4e4a-bbdb-420bad31388e", "id": 2}] 2024-03-29T11:20:50.623 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.646+0000 7f9fe8814640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:20:50.623 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.646+0000 7f9fe8814640 1 -- 172.21.15.176:0/3259740371 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd new", "uuid": "d7e8c409-bc66-4e4a-bbdb-420bad31388e", "id": 2} v 0) v1 -- 0x7f9fc80058b0 con 0x7f9fe409ac40 2024-03-29T11:20:50.627 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.650+0000 7f9fdaffd640 1 -- 172.21.15.176:0/3259740371 <== mon.1 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d7e8c409-bc66-4e4a-bbdb-420bad31388e", "id": 2}]=0 v5) v1 ==== 112+0+2 (unknown 1354531602 0 216599125) 0x7f9fd4006120 con 0x7f9fe409ac40 2024-03-29T11:20:50.627 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.650+0000 7f9fdaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "d7e8c409-bc66-4e4a-bbdb-420bad31388e", "id": 2}] 2024-03-29T11:20:50.627 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.650+0000 7f9fdaffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:50.627 INFO:teuthology.orchestra.run.smithi176.stdout:2 2024-03-29T11:20:50.629 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.651+0000 7f9fe8814640 10 librados: watch_flush enter 2024-03-29T11:20:50.629 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.651+0000 7f9fe8814640 10 librados: watch_flush exit 2024-03-29T11:20:50.629 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.651+0000 7f9fe8814640 20 client.4123.objecter shutdown clearing up homeless session... 2024-03-29T11:20:50.629 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.651+0000 7f9fe8814640 10 client.4123.objecter successfully canceled tick 2024-03-29T11:20:50.629 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.651+0000 7f9fe8814640 10 monclient: shutdown 2024-03-29T11:20:50.629 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.651+0000 7f9fe8814640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:50.629 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.651+0000 7f9fe8814640 1 -- 172.21.15.176:0/3259740371 >> v1:172.21.15.176:6789/0 conn(0x7f9fe409ac40 legacy=0x7f9fe4143710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:50.629 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.651+0000 7f9fe8814640 1 -- 172.21.15.176:0/3259740371 shutdown_connections 2024-03-29T11:20:50.629 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.651+0000 7f9fe8814640 1 -- 172.21.15.176:0/3259740371 >> 172.21.15.176:0/3259740371 conn(0x7f9fe4066b00 msgr2=0x7f9fe413a2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:50.629 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.652+0000 7f9fe8814640 1 -- 172.21.15.176:0/3259740371 shutdown_connections 2024-03-29T11:20:50.629 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.652+0000 7f9fe8814640 1 -- 172.21.15.176:0/3259740371 wait complete. 2024-03-29T11:20:50.629 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.652+0000 7f9fe8814640 1 librados: shutdown 2024-03-29T11:20:50.638 DEBUG:teuthology.orchestra.run.smithi176:> sudo ceph --cluster ceph osd new a0682989-3595-4cd6-90a5-ab9111c278e7 3 2024-03-29T11:20:50.643 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-03-29T11:20:50.667+0000 7fedaa420040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T11:20:50.655 INFO:tasks.ceph.mgr.y.smithi114.stderr:2024-03-29T11:20:50.678+0000 7ffbece52040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T11:20:50.697 INFO:tasks.ceph.mgr.y.smithi114.stderr:2024-03-29T11:20:50.721+0000 7ffbece52040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T11:20:50.700 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-03-29T11:20:50.724+0000 7fedaa420040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T11:20:50.738 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.762+0000 7fd0790ff640 10 monclient: get_monmap_and_config 2024-03-29T11:20:50.738 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.762+0000 7fd0790ff640 10 monclient: build_initial_monmap 2024-03-29T11:20:50.738 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.762+0000 7fd0790ff640 10 monclient: monmap: 2024-03-29T11:20:50.738 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 0 2024-03-29T11:20:50.738 INFO:teuthology.orchestra.run.smithi176.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:20:50.739 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-29T11:20:50.763146+0000 2024-03-29T11:20:50.739 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-29T11:20:50.763146+0000 2024-03-29T11:20:50.739 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:50.739 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T11:20:50.739 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:20:50.739 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T11:20:50.739 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T11:20:50.739 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T11:20:50.747 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.762+0000 7fd0790ff640 1 Processor -- start 2024-03-29T11:20:50.747 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.762+0000 7fd0790ff640 1 -- start start 2024-03-29T11:20:50.747 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.762+0000 7fd0790ff640 10 monclient: init 2024-03-29T11:20:50.747 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.763+0000 7fd0790ff640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:50.747 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.763+0000 7fd0790ff640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:20:50.747 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.763+0000 7fd0790ff640 10 monclient(hunting): picked mon.noname-c con 0x7fd0740b8240 addr v1:172.21.15.176:6789/0 2024-03-29T11:20:50.747 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.763+0000 7fd0790ff640 10 monclient(hunting): picked mon.noname-b con 0x7fd0740b1f60 addr v1:172.21.15.114:6790/0 2024-03-29T11:20:50.747 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.763+0000 7fd0790ff640 10 monclient(hunting): picked mon.noname-a con 0x7fd07409c8c0 addr v1:172.21.15.114:6789/0 2024-03-29T11:20:50.748 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.763+0000 7fd0790ff640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd07409bd80 con 0x7fd07409c8c0 2024-03-29T11:20:50.748 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.763+0000 7fd0790ff640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd07409b960 con 0x7fd0740b8240 2024-03-29T11:20:50.748 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.763+0000 7fd0790ff640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0740b2ec0 con 0x7fd0740b1f60 2024-03-29T11:20:50.748 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.763+0000 7fd0790ff640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:50.748 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.763+0000 7fd0790ff640 10 monclient(hunting): authenticate will time out at 916.104672s 2024-03-29T11:20:50.748 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.763+0000 7fd072d9d640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7fd0740b8240 0x7fd07413f450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.176:49052/0 (socket says 172.21.15.176:49052) 2024-03-29T11:20:50.748 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.763+0000 7fd072d9d640 1 -- 172.21.15.176:0/147226803 learned_addr learned my addr 172.21.15.176:0/147226803 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-29T11:20:50.748 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.763+0000 7fd07259c640 1 -- 172.21.15.176:0/147226803 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1797320833 0 0) 0x7fd07409b960 con 0x7fd0740b8240 2024-03-29T11:20:50.748 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.763+0000 7fd07259c640 10 monclient(hunting): _negotiate 2024-03-29T11:20:50.748 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.763+0000 7fd07259c640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:50.748 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.763+0000 7fd07259c640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:50.748 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.763+0000 7fd07259c640 10 monclient(hunting): my global_id is 4126 2024-03-29T11:20:50.748 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.763+0000 7fd07259c640 1 -- 172.21.15.176:0/147226803 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd060003520 con 0x7fd0740b8240 2024-03-29T11:20:50.748 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.763+0000 7fd07259c640 1 -- 172.21.15.176:0/147226803 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 375559867 0 0) 0x7fd07409bd80 con 0x7fd07409c8c0 2024-03-29T11:20:50.748 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.763+0000 7fd07259c640 10 monclient(hunting): _negotiate 2024-03-29T11:20:50.749 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.763+0000 7fd07259c640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:50.749 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.763+0000 7fd07259c640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:50.749 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.763+0000 7fd07259c640 10 monclient(hunting): my global_id is 4131 2024-03-29T11:20:50.749 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.763+0000 7fd07259c640 1 -- 172.21.15.176:0/147226803 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd07409b960 con 0x7fd07409c8c0 2024-03-29T11:20:50.749 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.763+0000 7fd07259c640 1 -- 172.21.15.176:0/147226803 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3240375516 0 0) 0x7fd0740b2ec0 con 0x7fd0740b1f60 2024-03-29T11:20:50.749 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.763+0000 7fd07259c640 10 monclient(hunting): _negotiate 2024-03-29T11:20:50.749 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.763+0000 7fd07259c640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:50.749 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.763+0000 7fd07259c640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:50.749 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.763+0000 7fd07259c640 10 monclient(hunting): my global_id is 4121 2024-03-29T11:20:50.749 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.763+0000 7fd07259c640 1 -- 172.21.15.176:0/147226803 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd07409bd80 con 0x7fd0740b1f60 2024-03-29T11:20:50.749 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.764+0000 7fd07259c640 1 -- 172.21.15.176:0/147226803 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2647569555 0 0) 0x7fd060003520 con 0x7fd0740b8240 2024-03-29T11:20:50.749 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.764+0000 7fd07259c640 1 -- 172.21.15.176:0/147226803 >> v1:172.21.15.114:6790/0 conn(0x7fd0740b1f60 legacy=0x7fd07409c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:50.749 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.764+0000 7fd07259c640 1 -- 172.21.15.176:0/147226803 >> v1:172.21.15.114:6789/0 conn(0x7fd07409c8c0 legacy=0x7fd07414d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:50.749 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.764+0000 7fd07259c640 10 monclient: _finish_hunting 0 2024-03-29T11:20:50.749 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.764+0000 7fd07259c640 1 monclient: found mon.noname-c 2024-03-29T11:20:50.750 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.764+0000 7fd07259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:50.750 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.764+0000 7fd07259c640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.176:6789/0 2024-03-29T11:20:50.750 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.764+0000 7fd07259c640 1 -- 172.21.15.176:0/147226803 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd07413fb60 con 0x7fd0740b8240 2024-03-29T11:20:50.750 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.764+0000 7fd07259c640 10 monclient: _finish_auth 0 2024-03-29T11:20:50.750 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.764+0000 7fd07259c640 10 monclient: _check_auth_tickets 2024-03-29T11:20:50.750 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.764+0000 7fd07259c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:50.750 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.764+0000 7fd0790ff640 5 monclient: authenticate success, global_id 4126 2024-03-29T11:20:50.750 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.764+0000 7fd07259c640 1 -- 172.21.15.176:0/147226803 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fd064001690 con 0x7fd0740b8240 2024-03-29T11:20:50.750 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.764+0000 7fd0790ff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:50.750 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.764+0000 7fd07259c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:50.750 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.764+0000 7fd07259c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:20:50.750 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.764+0000 7fd07259c640 10 monclient: dump: 2024-03-29T11:20:50.750 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-03-29T11:20:50.751 INFO:teuthology.orchestra.run.smithi176.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:20:50.751 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:50.751 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:50.751 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:50.751 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T11:20:50.751 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:20:50.751 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:20:50.751 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:20:50.751 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T11:20:50.751 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.764+0000 7fd0790ff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:50.751 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.764+0000 7fd07259c640 1 -- 172.21.15.176:0/147226803 <== mon.1 v1:172.21.15.176:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd064001870 con 0x7fd0740b8240 2024-03-29T11:20:50.751 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.764+0000 7fd07259c640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:50.751 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.764+0000 7fd07259c640 1 -- 172.21.15.176:0/147226803 <== mon.1 v1:172.21.15.176:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fd064001cc0 con 0x7fd0740b8240 2024-03-29T11:20:50.751 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.764+0000 7fd0790ff640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:50.751 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.764+0000 7fd0790ff640 4 set_mon_vals no callback set 2024-03-29T11:20:50.752 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.764+0000 7fd07259c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:50.752 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.764+0000 7fd07259c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:20:50.752 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.764+0000 7fd07259c640 10 monclient: dump: 2024-03-29T11:20:50.752 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-03-29T11:20:50.752 INFO:teuthology.orchestra.run.smithi176.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:20:50.752 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:50.752 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:50.752 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:50.752 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T11:20:50.752 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:20:50.752 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:20:50.752 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:20:50.752 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T11:20:50.752 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.764+0000 7fd0790ff640 10 monclient: shutdown 2024-03-29T11:20:50.753 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.764+0000 7fd0790ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:50.753 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.764+0000 7fd0790ff640 1 -- 172.21.15.176:0/147226803 >> v1:172.21.15.176:6789/0 conn(0x7fd0740b8240 legacy=0x7fd07413f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:50.753 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.764+0000 7fd0790ff640 1 -- 172.21.15.176:0/147226803 shutdown_connections 2024-03-29T11:20:50.753 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.764+0000 7fd0790ff640 1 -- 172.21.15.176:0/147226803 >> 172.21.15.176:0/147226803 conn(0x7fd074066b00 msgr2=0x7fd074066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:50.753 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.764+0000 7fd0790ff640 1 -- 172.21.15.176:0/147226803 shutdown_connections 2024-03-29T11:20:50.753 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.764+0000 7fd0790ff640 1 -- 172.21.15.176:0/147226803 wait complete. 2024-03-29T11:20:50.753 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.765+0000 7fd0790ff640 10 monclient: build_initial_monmap 2024-03-29T11:20:50.753 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.765+0000 7fd0790ff640 10 monclient: monmap: 2024-03-29T11:20:50.753 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 0 2024-03-29T11:20:50.753 INFO:teuthology.orchestra.run.smithi176.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:50.753 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 0.000000 2024-03-29T11:20:50.753 INFO:teuthology.orchestra.run.smithi176.stderr:created 0.000000 2024-03-29T11:20:50.753 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:50.753 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T11:20:50.754 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:20:50.754 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T11:20:50.754 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T11:20:50.754 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T11:20:50.754 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.765+0000 7fd0790ff640 1 librados: starting msgr at 2024-03-29T11:20:50.754 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.765+0000 7fd0790ff640 1 librados: starting objecter 2024-03-29T11:20:50.754 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.766+0000 7fd0790ff640 1 Processor -- start 2024-03-29T11:20:50.754 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.766+0000 7fd0790ff640 1 -- start start 2024-03-29T11:20:50.754 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.766+0000 7fd0790ff640 1 librados: setting wanted keys 2024-03-29T11:20:50.754 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.766+0000 7fd0790ff640 1 librados: calling monclient init 2024-03-29T11:20:50.754 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.766+0000 7fd0790ff640 10 monclient: init 2024-03-29T11:20:50.754 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.766+0000 7fd0790ff640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:50.754 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.766+0000 7fd0790ff640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:20:50.754 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.766+0000 7fd0790ff640 10 monclient(hunting): picked mon.noname-a con 0x7fd07409c8c0 addr v1:172.21.15.114:6789/0 2024-03-29T11:20:50.754 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.766+0000 7fd0790ff640 10 monclient(hunting): picked mon.noname-c con 0x7fd0740b8240 addr v1:172.21.15.114:6790/0 2024-03-29T11:20:50.755 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.766+0000 7fd0790ff640 10 monclient(hunting): picked mon.noname-b con 0x7fd074132460 addr v1:172.21.15.176:6789/0 2024-03-29T11:20:50.755 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.766+0000 7fd0790ff640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0741d7d50 con 0x7fd07409c8c0 2024-03-29T11:20:50.755 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.766+0000 7fd0790ff640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0741d7910 con 0x7fd074132460 2024-03-29T11:20:50.755 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.766+0000 7fd0790ff640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0741d7aa0 con 0x7fd0740b8240 2024-03-29T11:20:50.755 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.766+0000 7fd0790ff640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:50.755 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.766+0000 7fd0790ff640 10 monclient(hunting): authenticate will time out at 916.107889s 2024-03-29T11:20:50.755 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.766+0000 7fd07359e640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7fd074132460 0x7fd074142060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.176:49058/0 (socket says 172.21.15.176:49058) 2024-03-29T11:20:50.755 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.766+0000 7fd07359e640 1 -- 172.21.15.176:0/822331093 learned_addr learned my addr 172.21.15.176:0/822331093 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-29T11:20:50.755 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.766+0000 7fd06b7fe640 10 client.?.objecter ms_handle_connect 0x7fd07409c8c0 2024-03-29T11:20:50.755 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.766+0000 7fd06b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:50.755 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.766+0000 7fd06b7fe640 10 client.?.objecter ms_handle_connect 0x7fd074132460 2024-03-29T11:20:50.755 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.766+0000 7fd06b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:50.755 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.766+0000 7fd06b7fe640 10 client.?.objecter ms_handle_connect 0x7fd0740b8240 2024-03-29T11:20:50.755 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.766+0000 7fd06b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:50.756 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.767+0000 7fd06b7fe640 1 -- 172.21.15.176:0/822331093 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 994727551 0 0) 0x7fd0741d7d50 con 0x7fd07409c8c0 2024-03-29T11:20:50.756 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.767+0000 7fd06b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:20:50.756 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.767+0000 7fd06b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:50.756 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.767+0000 7fd06b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:50.756 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.767+0000 7fd06b7fe640 10 monclient(hunting): my global_id is 4134 2024-03-29T11:20:50.756 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.767+0000 7fd06b7fe640 1 -- 172.21.15.176:0/822331093 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd054004eb0 con 0x7fd07409c8c0 2024-03-29T11:20:50.756 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.767+0000 7fd06b7fe640 1 -- 172.21.15.176:0/822331093 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4128884328 0 0) 0x7fd0741d7910 con 0x7fd074132460 2024-03-29T11:20:50.756 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.767+0000 7fd06b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:20:50.756 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.767+0000 7fd06b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:50.756 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.767+0000 7fd06b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:50.756 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.767+0000 7fd06b7fe640 10 monclient(hunting): my global_id is 4129 2024-03-29T11:20:50.756 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.767+0000 7fd06b7fe640 1 -- 172.21.15.176:0/822331093 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd0741d7d50 con 0x7fd074132460 2024-03-29T11:20:50.756 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.767+0000 7fd06b7fe640 1 -- 172.21.15.176:0/822331093 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1564921930 0 0) 0x7fd0741d7aa0 con 0x7fd0740b8240 2024-03-29T11:20:50.756 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.767+0000 7fd06b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:20:50.756 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.767+0000 7fd06b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:50.757 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.767+0000 7fd06b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:50.757 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.767+0000 7fd06b7fe640 10 monclient(hunting): my global_id is 4124 2024-03-29T11:20:50.757 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.767+0000 7fd06b7fe640 1 -- 172.21.15.176:0/822331093 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd0741d7910 con 0x7fd0740b8240 2024-03-29T11:20:50.757 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.767+0000 7fd06b7fe640 1 -- 172.21.15.176:0/822331093 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4000817220 0 0) 0x7fd054004eb0 con 0x7fd07409c8c0 2024-03-29T11:20:50.757 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.767+0000 7fd06b7fe640 1 -- 172.21.15.176:0/822331093 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd0741d7aa0 con 0x7fd07409c8c0 2024-03-29T11:20:50.757 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.767+0000 7fd06b7fe640 1 -- 172.21.15.176:0/822331093 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fd064001bc0 con 0x7fd07409c8c0 2024-03-29T11:20:50.757 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.767+0000 7fd06b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:50.757 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.767+0000 7fd06b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:20:50.757 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.767+0000 7fd06b7fe640 10 monclient(hunting): dump: 2024-03-29T11:20:50.757 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-03-29T11:20:50.757 INFO:teuthology.orchestra.run.smithi176.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:20:50.757 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:50.757 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:50.757 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:50.758 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T11:20:50.758 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:20:50.758 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:20:50.758 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:20:50.758 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T11:20:50.758 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.767+0000 7fd06b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:50.758 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.767+0000 7fd06b7fe640 1 -- 172.21.15.176:0/822331093 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1210208597 0 0) 0x7fd0741d7910 con 0x7fd0740b8240 2024-03-29T11:20:50.758 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.767+0000 7fd06b7fe640 1 -- 172.21.15.176:0/822331093 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd054004eb0 con 0x7fd0740b8240 2024-03-29T11:20:50.758 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.767+0000 7fd06b7fe640 1 -- 172.21.15.176:0/822331093 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fd05c001910 con 0x7fd0740b8240 2024-03-29T11:20:50.758 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.767+0000 7fd06b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:50.758 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.767+0000 7fd06b7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:20:50.758 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.767+0000 7fd06b7fe640 10 monclient(hunting): dump: 2024-03-29T11:20:50.758 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-03-29T11:20:50.758 INFO:teuthology.orchestra.run.smithi176.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:20:50.758 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:50.758 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:50.759 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:50.759 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T11:20:50.759 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:20:50.759 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:20:50.759 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:20:50.759 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T11:20:50.759 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.767+0000 7fd06b7fe640 1 -- 172.21.15.176:0/822331093 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3984021918 0 0) 0x7fd0741d7d50 con 0x7fd074132460 2024-03-29T11:20:50.759 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.767+0000 7fd06b7fe640 1 -- 172.21.15.176:0/822331093 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd0741d7910 con 0x7fd074132460 2024-03-29T11:20:50.759 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.767+0000 7fd06b7fe640 1 -- 172.21.15.176:0/822331093 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fd06c001a30 con 0x7fd074132460 2024-03-29T11:20:50.759 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.767+0000 7fd06b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:50.759 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.767+0000 7fd06b7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:20:50.759 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.767+0000 7fd06b7fe640 10 monclient(hunting): dump: 2024-03-29T11:20:50.759 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-03-29T11:20:50.759 INFO:teuthology.orchestra.run.smithi176.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:20:50.759 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:50.760 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:50.760 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:50.760 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T11:20:50.760 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:20:50.760 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:20:50.760 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:20:50.760 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T11:20:50.760 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.767+0000 7fd06b7fe640 1 -- 172.21.15.176:0/822331093 <== mon.0 v1:172.21.15.114:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 649855622 0 0) 0x7fd0741d7aa0 con 0x7fd07409c8c0 2024-03-29T11:20:50.760 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.767+0000 7fd06b7fe640 1 -- 172.21.15.176:0/822331093 >> v1:172.21.15.114:6790/0 conn(0x7fd0740b8240 legacy=0x7fd074141950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:50.760 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.767+0000 7fd06b7fe640 1 -- 172.21.15.176:0/822331093 >> v1:172.21.15.176:6789/0 conn(0x7fd074132460 legacy=0x7fd074142060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:50.760 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.767+0000 7fd06b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:20:50.760 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.767+0000 7fd06b7fe640 1 monclient: found mon.a 2024-03-29T11:20:50.760 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.767+0000 7fd06b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:50.760 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.767+0000 7fd06b7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T11:20:50.760 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.767+0000 7fd06b7fe640 1 -- 172.21.15.176:0/822331093 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0741e5c80 con 0x7fd07409c8c0 2024-03-29T11:20:50.761 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.767+0000 7fd06b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:20:50.761 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.767+0000 7fd06b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:20:50.761 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.767+0000 7fd06b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:50.761 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.767+0000 7fd0790ff640 5 monclient: authenticate success, global_id 4134 2024-03-29T11:20:50.761 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.767+0000 7fd0790ff640 10 monclient: _renew_subs 2024-03-29T11:20:50.761 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.767+0000 7fd0790ff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T11:20:50.761 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.767+0000 7fd0790ff640 1 -- 172.21.15.176:0/822331093 --> v1:172.21.15.114:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0741e2c50 con 0x7fd07409c8c0 2024-03-29T11:20:50.761 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.767+0000 7fd0790ff640 10 client.4134.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:50.761 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.767+0000 7fd0790ff640 10 monclient: _renew_subs 2024-03-29T11:20:50.761 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.767+0000 7fd0790ff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T11:20:50.761 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.767+0000 7fd0790ff640 1 -- 172.21.15.176:0/822331093 --> v1:172.21.15.114:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0741e31d0 con 0x7fd07409c8c0 2024-03-29T11:20:50.761 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.767+0000 7fd0790ff640 1 librados: init done 2024-03-29T11:20:50.761 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.767+0000 7fd06b7fe640 1 -- 172.21.15.176:0/822331093 <== mon.0 v1:172.21.15.114:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd064004a90 con 0x7fd07409c8c0 2024-03-29T11:20:50.761 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.767+0000 7fd06b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:50.761 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.767+0000 7fd06b7fe640 1 -- 172.21.15.176:0/822331093 <== mon.0 v1:172.21.15.114:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fd064004da0 con 0x7fd07409c8c0 2024-03-29T11:20:50.761 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.767+0000 7fd06b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:50.762 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.767+0000 7fd071d9b640 4 set_mon_vals no callback set 2024-03-29T11:20:50.762 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.767+0000 7fd06b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:20:50.762 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.767+0000 7fd06b7fe640 10 monclient: dump: 2024-03-29T11:20:50.762 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-03-29T11:20:50.762 INFO:teuthology.orchestra.run.smithi176.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:20:50.762 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:50.762 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:50.762 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:50.762 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T11:20:50.762 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:20:50.762 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:20:50.762 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:20:50.762 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T11:20:50.762 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.767+0000 7fd06b7fe640 1 -- 172.21.15.176:0/822331093 <== mon.0 v1:172.21.15.114:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fd064005290 con 0x7fd07409c8c0 2024-03-29T11:20:50.762 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.767+0000 7fd06b7fe640 1 -- 172.21.15.176:0/822331093 <== mon.0 v1:172.21.15.114:6789/0 8 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (unknown 3878468697 0 0) 0x7fd0640067c0 con 0x7fd07409c8c0 2024-03-29T11:20:50.763 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.767+0000 7fd06b7fe640 10 client.4134.objecter ms_dispatch 0x7fd074000b90 osd_map(5..5 src has 1..5) v4 2024-03-29T11:20:50.763 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.767+0000 7fd06b7fe640 3 client.4134.objecter handle_osd_map got epochs [5,5] > 0 2024-03-29T11:20:50.763 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.767+0000 7fd06b7fe640 3 client.4134.objecter handle_osd_map decoding full epoch 5 2024-03-29T11:20:50.763 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.767+0000 7fd06b7fe640 20 client.4134.objecter dump_active .. 0 homeless 2024-03-29T11:20:50.763 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.768+0000 7fd0790ff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:50.763 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.768+0000 7fd0790ff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:50.763 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.768+0000 7fd0790ff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T11:20:50.763 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.768+0000 7fd0790ff640 1 -- 172.21.15.176:0/822331093 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd058005080 con 0x7fd07409c8c0 2024-03-29T11:20:50.763 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-03-29T11:20:50.770+0000 7fedaa420040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T11:20:50.764 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.770+0000 7fd06b7fe640 1 -- 172.21.15.176:0/822331093 <== mon.0 v1:172.21.15.114:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fd0640063e0 con 0x7fd07409c8c0 2024-03-29T11:20:50.764 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.770+0000 7fd06b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:50.764 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.770+0000 7fd06b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:50.813 INFO:tasks.ceph.mgr.y.smithi114.stderr:2024-03-29T11:20:50.836+0000 7ffbece52040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T11:20:50.819 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.843+0000 7fd0790ff640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "a0682989-3595-4cd6-90a5-ab9111c278e7", "id": 3}] 2024-03-29T11:20:50.819 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.843+0000 7fd0790ff640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "a0682989-3595-4cd6-90a5-ab9111c278e7", "id": 3}] 2024-03-29T11:20:50.820 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.843+0000 7fd0790ff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T11:20:50.820 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.843+0000 7fd0790ff640 1 -- 172.21.15.176:0/822331093 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "osd new", "uuid": "a0682989-3595-4cd6-90a5-ab9111c278e7", "id": 3} v 0) v1 -- 0x7fd0580058b0 con 0x7fd07409c8c0 2024-03-29T11:20:50.821 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.845+0000 7fd06b7fe640 1 -- 172.21.15.176:0/822331093 <== mon.0 v1:172.21.15.114:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "a0682989-3595-4cd6-90a5-ab9111c278e7", "id": 3}]=0 v6) v1 ==== 112+0+2 (unknown 3382182755 0 525046306) 0x7fd064006980 con 0x7fd07409c8c0 2024-03-29T11:20:50.821 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.845+0000 7fd06b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "a0682989-3595-4cd6-90a5-ab9111c278e7", "id": 3}] 2024-03-29T11:20:50.821 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.845+0000 7fd06b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:50.822 INFO:teuthology.orchestra.run.smithi176.stdout:3 2024-03-29T11:20:50.823 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.845+0000 7fd0790ff640 10 librados: watch_flush enter 2024-03-29T11:20:50.823 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.845+0000 7fd0790ff640 10 librados: watch_flush exit 2024-03-29T11:20:50.823 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.845+0000 7fd0790ff640 20 client.4134.objecter shutdown clearing up homeless session... 2024-03-29T11:20:50.823 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.845+0000 7fd0790ff640 10 client.4134.objecter successfully canceled tick 2024-03-29T11:20:50.823 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.845+0000 7fd0790ff640 10 monclient: shutdown 2024-03-29T11:20:50.823 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.845+0000 7fd0790ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:50.823 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.845+0000 7fd0790ff640 1 -- 172.21.15.176:0/822331093 >> v1:172.21.15.114:6789/0 conn(0x7fd07409c8c0 legacy=0x7fd0741d7200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:50.823 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.846+0000 7fd0790ff640 1 -- 172.21.15.176:0/822331093 shutdown_connections 2024-03-29T11:20:50.823 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.846+0000 7fd0790ff640 1 -- 172.21.15.176:0/822331093 >> 172.21.15.176:0/822331093 conn(0x7fd074066b00 msgr2=0x7fd07413da10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:50.823 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.846+0000 7fd0790ff640 1 -- 172.21.15.176:0/822331093 shutdown_connections 2024-03-29T11:20:50.824 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.846+0000 7fd0790ff640 1 -- 172.21.15.176:0/822331093 wait complete. 2024-03-29T11:20:50.824 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.846+0000 7fd0790ff640 1 librados: shutdown 2024-03-29T11:20:50.832 DEBUG:teuthology.orchestra.run.smithi176:> sudo ceph --cluster ceph osd new 67068120-e3ff-4817-bd06-44feb7479a35 4 2024-03-29T11:20:50.867 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-03-29T11:20:50.891+0000 7fedaa420040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T11:20:50.884 INFO:tasks.ceph.mgr.y.smithi114.stderr:2024-03-29T11:20:50.907+0000 7ffbece52040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T11:20:50.930 INFO:tasks.ceph.mgr.y.smithi114.stderr:2024-03-29T11:20:50.953+0000 7ffbece52040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T11:20:50.941 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-03-29T11:20:50.964+0000 7fedaa420040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T11:20:50.967 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.990+0000 7ff402a4a640 10 monclient: get_monmap_and_config 2024-03-29T11:20:50.967 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.990+0000 7ff402a4a640 10 monclient: build_initial_monmap 2024-03-29T11:20:50.967 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.990+0000 7ff402a4a640 10 monclient: monmap: 2024-03-29T11:20:50.967 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 0 2024-03-29T11:20:50.967 INFO:teuthology.orchestra.run.smithi176.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:20:50.967 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-29T11:20:50.991688+0000 2024-03-29T11:20:50.967 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-29T11:20:50.991688+0000 2024-03-29T11:20:50.967 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:50.967 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T11:20:50.968 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:20:50.968 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T11:20:50.968 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T11:20:50.968 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T11:20:50.977 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.991+0000 7ff402a4a640 1 Processor -- start 2024-03-29T11:20:50.978 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.991+0000 7ff402a4a640 1 -- start start 2024-03-29T11:20:50.978 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.991+0000 7ff402a4a640 10 monclient: init 2024-03-29T11:20:50.978 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.991+0000 7ff402a4a640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:50.978 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.991+0000 7ff402a4a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:20:50.978 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.991+0000 7ff402a4a640 10 monclient(hunting): picked mon.noname-a con 0x7ff3fc09ac40 addr v1:172.21.15.114:6789/0 2024-03-29T11:20:50.978 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.991+0000 7ff402a4a640 10 monclient(hunting): picked mon.noname-b con 0x7ff3fc09c870 addr v1:172.21.15.114:6790/0 2024-03-29T11:20:50.978 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.991+0000 7ff402a4a640 10 monclient(hunting): picked mon.noname-c con 0x7ff3fc0b4360 addr v1:172.21.15.176:6789/0 2024-03-29T11:20:50.978 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.991+0000 7ff402a4a640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3fc09dec0 con 0x7ff3fc09ac40 2024-03-29T11:20:50.978 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.992+0000 7ff402a4a640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3fc09da30 con 0x7ff3fc0b4360 2024-03-29T11:20:50.978 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.992+0000 7ff402a4a640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3fc0b39c0 con 0x7ff3fc09c870 2024-03-29T11:20:50.978 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.992+0000 7ff402a4a640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:50.978 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.992+0000 7ff402a4a640 10 monclient(hunting): authenticate will time out at 916.333398s 2024-03-29T11:20:50.978 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.992+0000 7ff400fe7640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7ff3fc0b4360 0x7ff3fc14d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.176:49062/0 (socket says 172.21.15.176:49062) 2024-03-29T11:20:50.979 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.992+0000 7ff400fe7640 1 -- 172.21.15.176:0/595284206 learned_addr learned my addr 172.21.15.176:0/595284206 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-29T11:20:50.979 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.992+0000 7ff3faffd640 1 -- 172.21.15.176:0/595284206 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 544480674 0 0) 0x7ff3fc09da30 con 0x7ff3fc0b4360 2024-03-29T11:20:50.979 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.992+0000 7ff3faffd640 10 monclient(hunting): _negotiate 2024-03-29T11:20:50.979 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.992+0000 7ff3faffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:50.979 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.992+0000 7ff3faffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:50.979 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.992+0000 7ff3faffd640 10 monclient(hunting): my global_id is 4132 2024-03-29T11:20:50.979 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.992+0000 7ff3faffd640 1 -- 172.21.15.176:0/595284206 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3e4003520 con 0x7ff3fc0b4360 2024-03-29T11:20:50.979 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.992+0000 7ff3faffd640 1 -- 172.21.15.176:0/595284206 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1228569634 0 0) 0x7ff3f0001110 con 0x7ff3fc09ac40 2024-03-29T11:20:50.979 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.992+0000 7ff3faffd640 10 monclient(hunting): _negotiate 2024-03-29T11:20:50.979 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.992+0000 7ff3faffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:50.979 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.992+0000 7ff3faffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:50.979 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.992+0000 7ff3faffd640 10 monclient(hunting): my global_id is 4137 2024-03-29T11:20:50.979 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.992+0000 7ff3faffd640 1 -- 172.21.15.176:0/595284206 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3fc09da30 con 0x7ff3fc09ac40 2024-03-29T11:20:50.979 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.992+0000 7ff3faffd640 1 -- 172.21.15.176:0/595284206 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3499475352 0 0) 0x7ff3fc0b39c0 con 0x7ff3fc09c870 2024-03-29T11:20:50.980 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.992+0000 7ff3faffd640 10 monclient(hunting): _negotiate 2024-03-29T11:20:50.980 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.992+0000 7ff3faffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:50.980 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.992+0000 7ff3faffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:50.980 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.992+0000 7ff3faffd640 10 monclient(hunting): my global_id is 4127 2024-03-29T11:20:50.980 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.992+0000 7ff3faffd640 1 -- 172.21.15.176:0/595284206 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3f0001110 con 0x7ff3fc09c870 2024-03-29T11:20:50.980 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.992+0000 7ff3faffd640 1 -- 172.21.15.176:0/595284206 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3201145786 0 0) 0x7ff3fc09da30 con 0x7ff3fc09ac40 2024-03-29T11:20:50.980 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.992+0000 7ff3faffd640 1 -- 172.21.15.176:0/595284206 >> v1:172.21.15.114:6790/0 conn(0x7ff3fc09c870 legacy=0x7ff3fc13f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:50.980 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.992+0000 7ff3faffd640 1 -- 172.21.15.176:0/595284206 >> v1:172.21.15.176:6789/0 conn(0x7ff3fc0b4360 legacy=0x7ff3fc14d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:50.980 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.992+0000 7ff3faffd640 10 monclient: _finish_hunting 0 2024-03-29T11:20:50.980 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.992+0000 7ff3faffd640 1 monclient: found mon.noname-a 2024-03-29T11:20:50.980 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.992+0000 7ff3faffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:50.980 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.992+0000 7ff3faffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.114:6789/0 2024-03-29T11:20:50.980 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.992+0000 7ff3faffd640 1 -- 172.21.15.176:0/595284206 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3fc0b5160 con 0x7ff3fc09ac40 2024-03-29T11:20:50.980 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.992+0000 7ff3faffd640 10 monclient: _finish_auth 0 2024-03-29T11:20:50.980 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.992+0000 7ff3faffd640 10 monclient: _check_auth_tickets 2024-03-29T11:20:50.981 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.992+0000 7ff3faffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:50.981 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.992+0000 7ff402a4a640 5 monclient: authenticate success, global_id 4137 2024-03-29T11:20:50.981 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.992+0000 7ff3faffd640 1 -- 172.21.15.176:0/595284206 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7ff3f0001660 con 0x7ff3fc09ac40 2024-03-29T11:20:50.981 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.992+0000 7ff402a4a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:50.981 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.992+0000 7ff3faffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:50.981 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.992+0000 7ff3faffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:20:50.981 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.992+0000 7ff3faffd640 10 monclient: dump: 2024-03-29T11:20:50.981 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-03-29T11:20:50.981 INFO:teuthology.orchestra.run.smithi176.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:20:50.981 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:50.981 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:50.981 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:50.981 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T11:20:50.981 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:20:50.981 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:20:50.982 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:20:50.982 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T11:20:50.982 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.992+0000 7ff402a4a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:50.982 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.992+0000 7ff3faffd640 1 -- 172.21.15.176:0/595284206 <== mon.0 v1:172.21.15.114:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff3f00017c0 con 0x7ff3fc09ac40 2024-03-29T11:20:50.982 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.992+0000 7ff3faffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:50.982 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.992+0000 7ff402a4a640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:50.982 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.992+0000 7ff3faffd640 1 -- 172.21.15.176:0/595284206 <== mon.0 v1:172.21.15.114:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7ff3f0001c10 con 0x7ff3fc09ac40 2024-03-29T11:20:50.982 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.992+0000 7ff3faffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:50.982 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.992+0000 7ff402a4a640 4 set_mon_vals no callback set 2024-03-29T11:20:50.982 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.992+0000 7ff3faffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:20:50.982 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.992+0000 7ff3faffd640 10 monclient: dump: 2024-03-29T11:20:50.982 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-03-29T11:20:50.982 INFO:teuthology.orchestra.run.smithi176.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:20:50.982 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:50.983 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:50.983 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:50.983 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T11:20:50.983 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:20:50.983 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:20:50.983 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:20:50.983 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T11:20:50.983 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.993+0000 7ff402a4a640 10 monclient: shutdown 2024-03-29T11:20:50.983 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.993+0000 7ff402a4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:50.983 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.993+0000 7ff402a4a640 1 -- 172.21.15.176:0/595284206 >> v1:172.21.15.114:6789/0 conn(0x7ff3fc09ac40 legacy=0x7ff3fc0b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:50.983 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.993+0000 7ff402a4a640 1 -- 172.21.15.176:0/595284206 shutdown_connections 2024-03-29T11:20:50.983 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.993+0000 7ff402a4a640 1 -- 172.21.15.176:0/595284206 >> 172.21.15.176:0/595284206 conn(0x7ff3fc066b00 msgr2=0x7ff3fc066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:50.983 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.993+0000 7ff402a4a640 1 -- 172.21.15.176:0/595284206 shutdown_connections 2024-03-29T11:20:50.984 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.993+0000 7ff402a4a640 1 -- 172.21.15.176:0/595284206 wait complete. 2024-03-29T11:20:50.984 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.994+0000 7ff402a4a640 10 monclient: build_initial_monmap 2024-03-29T11:20:50.984 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.994+0000 7ff402a4a640 10 monclient: monmap: 2024-03-29T11:20:50.984 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 0 2024-03-29T11:20:50.984 INFO:teuthology.orchestra.run.smithi176.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:50.984 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 0.000000 2024-03-29T11:20:50.984 INFO:teuthology.orchestra.run.smithi176.stderr:created 0.000000 2024-03-29T11:20:50.984 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:50.984 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T11:20:50.984 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:20:50.984 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T11:20:50.984 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T11:20:50.984 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T11:20:50.984 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.994+0000 7ff402a4a640 1 librados: starting msgr at 2024-03-29T11:20:50.984 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.994+0000 7ff402a4a640 1 librados: starting objecter 2024-03-29T11:20:50.985 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.994+0000 7ff402a4a640 1 Processor -- start 2024-03-29T11:20:50.985 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.994+0000 7ff402a4a640 1 -- start start 2024-03-29T11:20:50.985 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.994+0000 7ff402a4a640 1 librados: setting wanted keys 2024-03-29T11:20:50.985 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.994+0000 7ff402a4a640 1 librados: calling monclient init 2024-03-29T11:20:50.985 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.994+0000 7ff402a4a640 10 monclient: init 2024-03-29T11:20:50.985 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.995+0000 7ff402a4a640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:50.985 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.995+0000 7ff402a4a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:20:50.985 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.995+0000 7ff402a4a640 10 monclient(hunting): picked mon.noname-a con 0x7ff3fc09ac40 addr v1:172.21.15.114:6789/0 2024-03-29T11:20:50.985 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.995+0000 7ff402a4a640 10 monclient(hunting): picked mon.noname-b con 0x7ff3fc1459c0 addr v1:172.21.15.176:6789/0 2024-03-29T11:20:50.985 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.995+0000 7ff402a4a640 10 monclient(hunting): picked mon.noname-c con 0x7ff3fc0b0bd0 addr v1:172.21.15.114:6790/0 2024-03-29T11:20:50.985 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.995+0000 7ff402a4a640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3fc0b06f0 con 0x7ff3fc09ac40 2024-03-29T11:20:50.985 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.995+0000 7ff402a4a640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3fc0b1060 con 0x7ff3fc1459c0 2024-03-29T11:20:50.985 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.995+0000 7ff402a4a640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3fc146460 con 0x7ff3fc0b0bd0 2024-03-29T11:20:50.986 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.995+0000 7ff402a4a640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:50.986 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.995+0000 7ff402a4a640 10 monclient(hunting): authenticate will time out at 916.336484s 2024-03-29T11:20:50.986 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.995+0000 7ff3fb7fe640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7ff3fc1459c0 0x7ff3fc1da230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.176:49076/0 (socket says 172.21.15.176:49076) 2024-03-29T11:20:50.986 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.995+0000 7ff3fb7fe640 1 -- 172.21.15.176:0/4251464794 learned_addr learned my addr 172.21.15.176:0/4251464794 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-29T11:20:50.986 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.995+0000 7ff3f8ff9640 10 client.?.objecter ms_handle_connect 0x7ff3fc1459c0 2024-03-29T11:20:50.986 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.995+0000 7ff3f8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:50.986 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.995+0000 7ff3f8ff9640 10 client.?.objecter ms_handle_connect 0x7ff3fc09ac40 2024-03-29T11:20:50.986 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.995+0000 7ff3f8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:50.986 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.995+0000 7ff3f8ff9640 10 client.?.objecter ms_handle_connect 0x7ff3fc0b0bd0 2024-03-29T11:20:50.986 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.995+0000 7ff3f8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:50.986 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.995+0000 7ff3f8ff9640 1 -- 172.21.15.176:0/4251464794 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 930715641 0 0) 0x7ff3fc0b1060 con 0x7ff3fc1459c0 2024-03-29T11:20:50.986 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.995+0000 7ff3f8ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:20:50.986 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.995+0000 7ff3f8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:50.986 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.995+0000 7ff3f8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:50.987 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.995+0000 7ff3f8ff9640 10 monclient(hunting): my global_id is 4135 2024-03-29T11:20:50.987 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.995+0000 7ff3f8ff9640 1 -- 172.21.15.176:0/4251464794 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3e8004eb0 con 0x7ff3fc1459c0 2024-03-29T11:20:50.987 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.995+0000 7ff3f8ff9640 1 -- 172.21.15.176:0/4251464794 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 856559361 0 0) 0x7ff3fc0b06f0 con 0x7ff3fc09ac40 2024-03-29T11:20:50.987 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.995+0000 7ff3f8ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:20:50.987 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.995+0000 7ff3f8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:50.987 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.995+0000 7ff3f8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:50.987 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.995+0000 7ff3f8ff9640 10 monclient(hunting): my global_id is 4140 2024-03-29T11:20:50.987 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.995+0000 7ff3f8ff9640 1 -- 172.21.15.176:0/4251464794 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3fc0b1060 con 0x7ff3fc09ac40 2024-03-29T11:20:50.987 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.995+0000 7ff3f8ff9640 1 -- 172.21.15.176:0/4251464794 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1231962219 0 0) 0x7ff3fc146460 con 0x7ff3fc0b0bd0 2024-03-29T11:20:50.987 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.995+0000 7ff3f8ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:20:50.987 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.995+0000 7ff3f8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:50.987 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.995+0000 7ff3f8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:50.987 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.995+0000 7ff3f8ff9640 10 monclient(hunting): my global_id is 4130 2024-03-29T11:20:50.987 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.995+0000 7ff3f8ff9640 1 -- 172.21.15.176:0/4251464794 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3fc0b06f0 con 0x7ff3fc0b0bd0 2024-03-29T11:20:50.987 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.995+0000 7ff3f8ff9640 1 -- 172.21.15.176:0/4251464794 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3429660798 0 0) 0x7ff3e8004eb0 con 0x7ff3fc1459c0 2024-03-29T11:20:50.988 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.995+0000 7ff3f8ff9640 1 -- 172.21.15.176:0/4251464794 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff3fc146460 con 0x7ff3fc1459c0 2024-03-29T11:20:50.988 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.995+0000 7ff3f8ff9640 1 -- 172.21.15.176:0/4251464794 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7ff3ec0019d0 con 0x7ff3fc1459c0 2024-03-29T11:20:50.988 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.995+0000 7ff3f8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:50.988 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.995+0000 7ff3f8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:20:50.988 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.995+0000 7ff3f8ff9640 10 monclient(hunting): dump: 2024-03-29T11:20:50.988 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-03-29T11:20:50.988 INFO:teuthology.orchestra.run.smithi176.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:20:50.988 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:50.988 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:50.988 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:50.988 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T11:20:50.988 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:20:50.988 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:20:50.988 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:20:50.989 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T11:20:50.989 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.995+0000 7ff3f8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:50.989 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.995+0000 7ff3f8ff9640 1 -- 172.21.15.176:0/4251464794 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2075358072 0 0) 0x7ff3fc0b1060 con 0x7ff3fc09ac40 2024-03-29T11:20:50.989 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.995+0000 7ff3f8ff9640 1 -- 172.21.15.176:0/4251464794 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff3e8004eb0 con 0x7ff3fc09ac40 2024-03-29T11:20:50.989 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.995+0000 7ff3f8ff9640 1 -- 172.21.15.176:0/4251464794 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7ff3f00011a0 con 0x7ff3fc09ac40 2024-03-29T11:20:50.989 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.995+0000 7ff3f8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:50.989 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.995+0000 7ff3f8ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:20:50.989 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.995+0000 7ff3f8ff9640 10 monclient(hunting): dump: 2024-03-29T11:20:50.989 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-03-29T11:20:50.989 INFO:teuthology.orchestra.run.smithi176.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:20:50.989 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:50.989 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:50.989 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:50.989 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T11:20:50.989 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:20:50.989 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:20:50.990 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:20:50.990 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T11:20:50.990 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.995+0000 7ff3f8ff9640 1 -- 172.21.15.176:0/4251464794 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4134137911 0 0) 0x7ff3fc0b06f0 con 0x7ff3fc0b0bd0 2024-03-29T11:20:50.990 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.996+0000 7ff3f8ff9640 1 -- 172.21.15.176:0/4251464794 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff3fc0b1060 con 0x7ff3fc0b0bd0 2024-03-29T11:20:50.990 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.996+0000 7ff3f8ff9640 1 -- 172.21.15.176:0/4251464794 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7ff3f4001c10 con 0x7ff3fc0b0bd0 2024-03-29T11:20:50.990 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.996+0000 7ff3f8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:50.990 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.996+0000 7ff3f8ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:20:50.990 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.996+0000 7ff3f8ff9640 10 monclient(hunting): dump: 2024-03-29T11:20:50.990 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-03-29T11:20:50.990 INFO:teuthology.orchestra.run.smithi176.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:20:50.990 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:50.990 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:50.990 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:50.990 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T11:20:50.991 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:20:50.991 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:20:50.991 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:20:50.991 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T11:20:50.991 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.996+0000 7ff3f8ff9640 1 -- 172.21.15.176:0/4251464794 <== mon.1 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2772523568 0 0) 0x7ff3fc146460 con 0x7ff3fc1459c0 2024-03-29T11:20:50.991 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.996+0000 7ff3f8ff9640 1 -- 172.21.15.176:0/4251464794 >> v1:172.21.15.114:6790/0 conn(0x7ff3fc0b0bd0 legacy=0x7ff3fc1e17b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:50.991 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.996+0000 7ff3f8ff9640 1 -- 172.21.15.176:0/4251464794 >> v1:172.21.15.114:6789/0 conn(0x7ff3fc09ac40 legacy=0x7ff3fc0afd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:50.991 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.996+0000 7ff3f8ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:20:50.991 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.996+0000 7ff3f8ff9640 1 monclient: found mon.b 2024-03-29T11:20:50.991 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.996+0000 7ff3f8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:50.991 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.996+0000 7ff3f8ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:20:50.991 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.996+0000 7ff3f8ff9640 1 -- 172.21.15.176:0/4251464794 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3fc1e5f00 con 0x7ff3fc1459c0 2024-03-29T11:20:50.991 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.996+0000 7ff3f8ff9640 10 monclient: _finish_auth 0 2024-03-29T11:20:50.991 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.996+0000 7ff3f8ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:20:50.992 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.996+0000 7ff3f8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:50.992 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.996+0000 7ff402a4a640 5 monclient: authenticate success, global_id 4135 2024-03-29T11:20:50.992 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.996+0000 7ff402a4a640 10 monclient: _renew_subs 2024-03-29T11:20:50.992 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.996+0000 7ff402a4a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:20:50.992 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.996+0000 7ff402a4a640 1 -- 172.21.15.176:0/4251464794 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3fc1e2ed0 con 0x7ff3fc1459c0 2024-03-29T11:20:50.992 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.996+0000 7ff402a4a640 10 client.4135.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:50.992 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.996+0000 7ff402a4a640 10 monclient: _renew_subs 2024-03-29T11:20:50.992 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.996+0000 7ff402a4a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:20:50.992 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.996+0000 7ff402a4a640 1 -- 172.21.15.176:0/4251464794 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3fc1e3450 con 0x7ff3fc1459c0 2024-03-29T11:20:50.992 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.996+0000 7ff402a4a640 1 librados: init done 2024-03-29T11:20:50.992 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.996+0000 7ff3f8ff9640 1 -- 172.21.15.176:0/4251464794 <== mon.1 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff3ec001c50 con 0x7ff3fc1459c0 2024-03-29T11:20:50.992 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.996+0000 7ff3f8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:50.992 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.996+0000 7ff3f8ff9640 1 -- 172.21.15.176:0/4251464794 <== mon.1 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7ff3ec003a40 con 0x7ff3fc1459c0 2024-03-29T11:20:50.992 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.996+0000 7ff3f8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:50.992 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.996+0000 7ff3f8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:20:50.993 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.996+0000 7ff3fa7fc640 4 set_mon_vals no callback set 2024-03-29T11:20:50.993 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.996+0000 7ff3f8ff9640 10 monclient: dump: 2024-03-29T11:20:50.993 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-03-29T11:20:50.993 INFO:teuthology.orchestra.run.smithi176.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:20:50.993 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:50.993 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:50.993 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:50.993 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T11:20:50.993 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:20:50.993 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:20:50.993 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:20:50.993 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T11:20:50.993 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.996+0000 7ff3f8ff9640 1 -- 172.21.15.176:0/4251464794 <== mon.1 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7ff3ec003fb0 con 0x7ff3fc1459c0 2024-03-29T11:20:50.993 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.996+0000 7ff3f8ff9640 1 -- 172.21.15.176:0/4251464794 <== mon.1 v1:172.21.15.176:6789/0 8 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (unknown 2161188551 0 0) 0x7ff3ec006750 con 0x7ff3fc1459c0 2024-03-29T11:20:50.993 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.996+0000 7ff3f8ff9640 10 client.4135.objecter ms_dispatch 0x7ff3fc000b90 osd_map(6..6 src has 1..6) v4 2024-03-29T11:20:50.994 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.996+0000 7ff3f8ff9640 3 client.4135.objecter handle_osd_map got epochs [6,6] > 0 2024-03-29T11:20:50.994 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.996+0000 7ff3f8ff9640 3 client.4135.objecter handle_osd_map decoding full epoch 6 2024-03-29T11:20:50.994 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.996+0000 7ff3f8ff9640 20 client.4135.objecter dump_active .. 0 homeless 2024-03-29T11:20:50.994 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.997+0000 7ff402a4a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:50.994 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.997+0000 7ff402a4a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:50.994 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.997+0000 7ff402a4a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:20:50.994 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.997+0000 7ff402a4a640 1 -- 172.21.15.176:0/4251464794 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3dc005080 con 0x7ff3fc1459c0 2024-03-29T11:20:50.994 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.999+0000 7ff3f8ff9640 1 -- 172.21.15.176:0/4251464794 <== mon.1 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7ff3ec004440 con 0x7ff3fc1459c0 2024-03-29T11:20:50.994 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.999+0000 7ff3f8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:50.994 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:50.999+0000 7ff3f8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:50.995 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-03-29T11:20:51.014+0000 7fedaa420040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T11:20:50.995 INFO:tasks.ceph.mgr.y.smithi114.stderr:2024-03-29T11:20:51.008+0000 7ffbece52040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T11:20:51.032 INFO:tasks.ceph.mgr.y.smithi114.stderr:2024-03-29T11:20:51.055+0000 7ffbece52040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T11:20:51.044 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-03-29T11:20:51.068+0000 7fedaa420040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T11:20:51.047 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.071+0000 7ff402a4a640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "67068120-e3ff-4817-bd06-44feb7479a35", "id": 4}] 2024-03-29T11:20:51.047 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.071+0000 7ff402a4a640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "67068120-e3ff-4817-bd06-44feb7479a35", "id": 4}] 2024-03-29T11:20:51.047 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.071+0000 7ff402a4a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:20:51.048 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.071+0000 7ff402a4a640 1 -- 172.21.15.176:0/4251464794 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd new", "uuid": "67068120-e3ff-4817-bd06-44feb7479a35", "id": 4} v 0) v1 -- 0x7ff3dc0058b0 con 0x7ff3fc1459c0 2024-03-29T11:20:51.051 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.074+0000 7ff3f8ff9640 1 -- 172.21.15.176:0/4251464794 <== mon.1 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "67068120-e3ff-4817-bd06-44feb7479a35", "id": 4}]=0 v7) v1 ==== 112+0+2 (unknown 2629548176 0 1697012583) 0x7ff3ec006240 con 0x7ff3fc1459c0 2024-03-29T11:20:51.051 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.074+0000 7ff3f8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "67068120-e3ff-4817-bd06-44feb7479a35", "id": 4}] 2024-03-29T11:20:51.051 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.074+0000 7ff3f8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:51.051 INFO:teuthology.orchestra.run.smithi176.stdout:4 2024-03-29T11:20:51.052 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.075+0000 7ff402a4a640 10 librados: watch_flush enter 2024-03-29T11:20:51.052 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.075+0000 7ff402a4a640 10 librados: watch_flush exit 2024-03-29T11:20:51.052 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.075+0000 7ff402a4a640 20 client.4135.objecter shutdown clearing up homeless session... 2024-03-29T11:20:51.053 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.075+0000 7ff402a4a640 10 client.4135.objecter successfully canceled tick 2024-03-29T11:20:51.053 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.075+0000 7ff402a4a640 10 monclient: shutdown 2024-03-29T11:20:51.053 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.075+0000 7ff402a4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:51.053 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.075+0000 7ff402a4a640 1 -- 172.21.15.176:0/4251464794 >> v1:172.21.15.176:6789/0 conn(0x7ff3fc1459c0 legacy=0x7ff3fc1da230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:51.053 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.075+0000 7ff402a4a640 1 -- 172.21.15.176:0/4251464794 shutdown_connections 2024-03-29T11:20:51.053 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.075+0000 7ff402a4a640 1 -- 172.21.15.176:0/4251464794 >> 172.21.15.176:0/4251464794 conn(0x7ff3fc066b00 msgr2=0x7ff3fc13e870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:51.053 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.075+0000 7ff402a4a640 1 -- 172.21.15.176:0/4251464794 shutdown_connections 2024-03-29T11:20:51.053 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.075+0000 7ff402a4a640 1 -- 172.21.15.176:0/4251464794 wait complete. 2024-03-29T11:20:51.053 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.075+0000 7ff402a4a640 1 librados: shutdown 2024-03-29T11:20:51.062 DEBUG:teuthology.orchestra.run.smithi176:> sudo ceph --cluster ceph osd new ecd09e8b-8a85-4997-bdea-75d754e6ee67 5 2024-03-29T11:20:51.095 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-03-29T11:20:51.119+0000 7fedaa420040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T11:20:51.162 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.185+0000 7f533cd7c640 10 monclient: get_monmap_and_config 2024-03-29T11:20:51.162 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.185+0000 7f533cd7c640 10 monclient: build_initial_monmap 2024-03-29T11:20:51.162 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.185+0000 7f533cd7c640 10 monclient: monmap: 2024-03-29T11:20:51.162 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 0 2024-03-29T11:20:51.162 INFO:teuthology.orchestra.run.smithi176.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:20:51.162 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-29T11:20:51.186562+0000 2024-03-29T11:20:51.162 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-29T11:20:51.186562+0000 2024-03-29T11:20:51.162 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:51.162 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T11:20:51.162 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:20:51.162 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T11:20:51.163 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T11:20:51.163 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T11:20:51.171 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.186+0000 7f533cd7c640 1 Processor -- start 2024-03-29T11:20:51.171 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.186+0000 7f533cd7c640 1 -- start start 2024-03-29T11:20:51.172 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.186+0000 7f533cd7c640 10 monclient: init 2024-03-29T11:20:51.172 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.186+0000 7f533cd7c640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:51.172 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.186+0000 7f533cd7c640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:20:51.172 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.186+0000 7f533cd7c640 10 monclient(hunting): picked mon.noname-c con 0x7f533809ac40 addr v1:172.21.15.176:6789/0 2024-03-29T11:20:51.172 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.186+0000 7f533cd7c640 10 monclient(hunting): picked mon.noname-a con 0x7f533809c870 addr v1:172.21.15.114:6789/0 2024-03-29T11:20:51.172 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.186+0000 7f533cd7c640 10 monclient(hunting): picked mon.noname-b con 0x7f53380b4360 addr v1:172.21.15.114:6790/0 2024-03-29T11:20:51.172 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.186+0000 7f533cd7c640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f533809dec0 con 0x7f533809c870 2024-03-29T11:20:51.172 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.186+0000 7f533cd7c640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f533809da30 con 0x7f533809ac40 2024-03-29T11:20:51.172 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.186+0000 7f533cd7c640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53380b39c0 con 0x7f53380b4360 2024-03-29T11:20:51.172 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.186+0000 7f533cd7c640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:51.172 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.186+0000 7f533cd7c640 10 monclient(hunting): authenticate will time out at 916.528122s 2024-03-29T11:20:51.172 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.187+0000 7f533659c640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f533809ac40 0x7f53380b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.176:49088/0 (socket says 172.21.15.176:49088) 2024-03-29T11:20:51.172 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.187+0000 7f533659c640 1 -- 172.21.15.176:0/3255296104 learned_addr learned my addr 172.21.15.176:0/3255296104 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-29T11:20:51.173 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.187+0000 7f5335d9b640 1 -- 172.21.15.176:0/3255296104 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4165988200 0 0) 0x7f533809da30 con 0x7f533809ac40 2024-03-29T11:20:51.173 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.187+0000 7f5335d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:20:51.173 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.187+0000 7f5335d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:51.173 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.187+0000 7f5335d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:51.173 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.187+0000 7f5335d9b640 10 monclient(hunting): my global_id is 4138 2024-03-29T11:20:51.173 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.187+0000 7f5335d9b640 1 -- 172.21.15.176:0/3255296104 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5324003520 con 0x7f533809ac40 2024-03-29T11:20:51.173 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.187+0000 7f5335d9b640 1 -- 172.21.15.176:0/3255296104 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 700284066 0 0) 0x7f53380b39c0 con 0x7f53380b4360 2024-03-29T11:20:51.173 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.187+0000 7f5335d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:20:51.173 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.187+0000 7f5335d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:51.173 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.187+0000 7f5335d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:51.173 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.187+0000 7f5335d9b640 10 monclient(hunting): my global_id is 4133 2024-03-29T11:20:51.173 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.187+0000 7f5335d9b640 1 -- 172.21.15.176:0/3255296104 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f533809da30 con 0x7f53380b4360 2024-03-29T11:20:51.173 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.187+0000 7f5335d9b640 1 -- 172.21.15.176:0/3255296104 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 450867165 0 0) 0x7f5320001110 con 0x7f533809c870 2024-03-29T11:20:51.173 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.187+0000 7f5335d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:20:51.173 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.187+0000 7f5335d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:51.173 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.187+0000 7f5335d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:51.174 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.187+0000 7f5335d9b640 10 monclient(hunting): my global_id is 4143 2024-03-29T11:20:51.174 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.187+0000 7f5335d9b640 1 -- 172.21.15.176:0/3255296104 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53380b39c0 con 0x7f533809c870 2024-03-29T11:20:51.174 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.187+0000 7f5335d9b640 1 -- 172.21.15.176:0/3255296104 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3363517859 0 0) 0x7f5324003520 con 0x7f533809ac40 2024-03-29T11:20:51.174 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.187+0000 7f5335d9b640 1 -- 172.21.15.176:0/3255296104 >> v1:172.21.15.114:6790/0 conn(0x7f53380b4360 legacy=0x7f533814d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:51.174 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.187+0000 7f5335d9b640 1 -- 172.21.15.176:0/3255296104 >> v1:172.21.15.114:6789/0 conn(0x7f533809c870 legacy=0x7f533813f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:51.174 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.187+0000 7f5335d9b640 10 monclient: _finish_hunting 0 2024-03-29T11:20:51.174 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.187+0000 7f5335d9b640 1 monclient: found mon.noname-c 2024-03-29T11:20:51.174 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.187+0000 7f5335d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:51.174 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.187+0000 7f5335d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.176:6789/0 2024-03-29T11:20:51.174 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.187+0000 7f5335d9b640 1 -- 172.21.15.176:0/3255296104 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53380b5160 con 0x7f533809ac40 2024-03-29T11:20:51.174 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.187+0000 7f5335d9b640 10 monclient: _finish_auth 0 2024-03-29T11:20:51.174 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.187+0000 7f5335d9b640 10 monclient: _check_auth_tickets 2024-03-29T11:20:51.174 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.187+0000 7f5335d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:51.174 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.187+0000 7f5335d9b640 1 -- 172.21.15.176:0/3255296104 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f5328001690 con 0x7f533809ac40 2024-03-29T11:20:51.174 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.187+0000 7f533cd7c640 5 monclient: authenticate success, global_id 4138 2024-03-29T11:20:51.175 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.187+0000 7f533cd7c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:51.175 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.187+0000 7f5335d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:51.175 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.187+0000 7f5335d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:20:51.175 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.187+0000 7f5335d9b640 10 monclient: dump: 2024-03-29T11:20:51.175 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-03-29T11:20:51.175 INFO:teuthology.orchestra.run.smithi176.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:20:51.175 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:51.175 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:51.175 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:51.175 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T11:20:51.175 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:20:51.175 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:20:51.175 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:20:51.175 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T11:20:51.175 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.187+0000 7f533cd7c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:51.176 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.187+0000 7f5335d9b640 1 -- 172.21.15.176:0/3255296104 <== mon.1 v1:172.21.15.176:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5328001870 con 0x7f533809ac40 2024-03-29T11:20:51.176 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.187+0000 7f5335d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:51.176 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.187+0000 7f533cd7c640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:51.176 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.187+0000 7f5335d9b640 1 -- 172.21.15.176:0/3255296104 <== mon.1 v1:172.21.15.176:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f5328001c80 con 0x7f533809ac40 2024-03-29T11:20:51.176 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.187+0000 7f5335d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:51.176 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.187+0000 7f533cd7c640 4 set_mon_vals no callback set 2024-03-29T11:20:51.176 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.187+0000 7f5335d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:20:51.176 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.187+0000 7f5335d9b640 10 monclient: dump: 2024-03-29T11:20:51.176 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-03-29T11:20:51.176 INFO:teuthology.orchestra.run.smithi176.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:20:51.176 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:51.176 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:51.176 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:51.176 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T11:20:51.176 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:20:51.177 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:20:51.177 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:20:51.177 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T11:20:51.177 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.188+0000 7f533cd7c640 10 monclient: shutdown 2024-03-29T11:20:51.177 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.188+0000 7f533cd7c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:51.177 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.188+0000 7f533cd7c640 1 -- 172.21.15.176:0/3255296104 >> v1:172.21.15.176:6789/0 conn(0x7f533809ac40 legacy=0x7f53380b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:51.177 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.188+0000 7f533cd7c640 1 -- 172.21.15.176:0/3255296104 shutdown_connections 2024-03-29T11:20:51.177 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.188+0000 7f533cd7c640 1 -- 172.21.15.176:0/3255296104 >> 172.21.15.176:0/3255296104 conn(0x7f5338066b00 msgr2=0x7f5338066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:51.177 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.188+0000 7f533cd7c640 1 -- 172.21.15.176:0/3255296104 shutdown_connections 2024-03-29T11:20:51.177 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.188+0000 7f533cd7c640 1 -- 172.21.15.176:0/3255296104 wait complete. 2024-03-29T11:20:51.177 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.189+0000 7f533cd7c640 10 monclient: build_initial_monmap 2024-03-29T11:20:51.177 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.189+0000 7f533cd7c640 10 monclient: monmap: 2024-03-29T11:20:51.177 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 0 2024-03-29T11:20:51.177 INFO:teuthology.orchestra.run.smithi176.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:51.178 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 0.000000 2024-03-29T11:20:51.178 INFO:teuthology.orchestra.run.smithi176.stderr:created 0.000000 2024-03-29T11:20:51.178 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:51.178 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T11:20:51.178 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:20:51.178 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T11:20:51.178 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T11:20:51.178 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T11:20:51.178 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.189+0000 7f533cd7c640 1 librados: starting msgr at 2024-03-29T11:20:51.178 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.189+0000 7f533cd7c640 1 librados: starting objecter 2024-03-29T11:20:51.178 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.189+0000 7f533cd7c640 1 Processor -- start 2024-03-29T11:20:51.178 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.189+0000 7f533cd7c640 1 -- start start 2024-03-29T11:20:51.178 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.189+0000 7f533cd7c640 1 librados: setting wanted keys 2024-03-29T11:20:51.178 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.189+0000 7f533cd7c640 1 librados: calling monclient init 2024-03-29T11:20:51.178 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.189+0000 7f533cd7c640 10 monclient: init 2024-03-29T11:20:51.179 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.189+0000 7f533cd7c640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:51.179 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.189+0000 7f533cd7c640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:20:51.179 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.190+0000 7f533cd7c640 10 monclient(hunting): picked mon.noname-b con 0x7f533809ac40 addr v1:172.21.15.176:6789/0 2024-03-29T11:20:51.179 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.190+0000 7f533cd7c640 10 monclient(hunting): picked mon.noname-c con 0x7f5338138f60 addr v1:172.21.15.114:6790/0 2024-03-29T11:20:51.179 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.190+0000 7f533cd7c640 10 monclient(hunting): picked mon.noname-a con 0x7f53380afdc0 addr v1:172.21.15.114:6789/0 2024-03-29T11:20:51.179 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.190+0000 7f533cd7c640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53381db610 con 0x7f53380afdc0 2024-03-29T11:20:51.179 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.190+0000 7f533cd7c640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5338139a00 con 0x7f533809ac40 2024-03-29T11:20:51.179 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.190+0000 7f533cd7c640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53380b0230 con 0x7f5338138f60 2024-03-29T11:20:51.179 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.190+0000 7f533cd7c640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:51.179 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.190+0000 7f533cd7c640 10 monclient(hunting): authenticate will time out at 916.531269s 2024-03-29T11:20:51.179 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.190+0000 7f533659c640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f533809ac40 0x7f53381dac80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.176:49092/0 (socket says 172.21.15.176:49092) 2024-03-29T11:20:51.179 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.190+0000 7f533659c640 1 -- 172.21.15.176:0/4206931501 learned_addr learned my addr 172.21.15.176:0/4206931501 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-29T11:20:51.179 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.190+0000 7f532effd640 10 client.?.objecter ms_handle_connect 0x7f533809ac40 2024-03-29T11:20:51.179 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.190+0000 7f532effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:51.179 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.190+0000 7f532effd640 10 client.?.objecter ms_handle_connect 0x7f53380afdc0 2024-03-29T11:20:51.180 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.190+0000 7f532effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:51.180 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.190+0000 7f532effd640 10 client.?.objecter ms_handle_connect 0x7f5338138f60 2024-03-29T11:20:51.180 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.190+0000 7f532effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:51.180 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.190+0000 7f532effd640 1 -- 172.21.15.176:0/4206931501 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2269122388 0 0) 0x7f5338139a00 con 0x7f533809ac40 2024-03-29T11:20:51.180 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.190+0000 7f532effd640 10 monclient(hunting): _negotiate 2024-03-29T11:20:51.180 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.190+0000 7f532effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:51.180 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.190+0000 7f532effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:51.180 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.190+0000 7f532effd640 10 monclient(hunting): my global_id is 4141 2024-03-29T11:20:51.180 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.190+0000 7f532effd640 1 -- 172.21.15.176:0/4206931501 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5318004eb0 con 0x7f533809ac40 2024-03-29T11:20:51.180 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.190+0000 7f532effd640 1 -- 172.21.15.176:0/4206931501 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3336389887 0 0) 0x7f53381db610 con 0x7f53380afdc0 2024-03-29T11:20:51.180 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.190+0000 7f532effd640 10 monclient(hunting): _negotiate 2024-03-29T11:20:51.180 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.190+0000 7f532effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:51.180 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.190+0000 7f532effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:51.180 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.190+0000 7f532effd640 10 monclient(hunting): my global_id is 4146 2024-03-29T11:20:51.181 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.190+0000 7f532effd640 1 -- 172.21.15.176:0/4206931501 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5338139a00 con 0x7f53380afdc0 2024-03-29T11:20:51.181 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.190+0000 7f532effd640 1 -- 172.21.15.176:0/4206931501 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3082159946 0 0) 0x7f53380b0230 con 0x7f5338138f60 2024-03-29T11:20:51.181 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.190+0000 7f532effd640 10 monclient(hunting): _negotiate 2024-03-29T11:20:51.181 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.190+0000 7f532effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:51.181 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.190+0000 7f532effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:51.181 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.190+0000 7f532effd640 10 monclient(hunting): my global_id is 4136 2024-03-29T11:20:51.181 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.190+0000 7f532effd640 1 -- 172.21.15.176:0/4206931501 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53381db610 con 0x7f5338138f60 2024-03-29T11:20:51.181 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.190+0000 7f532effd640 1 -- 172.21.15.176:0/4206931501 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1185528083 0 0) 0x7f5318004eb0 con 0x7f533809ac40 2024-03-29T11:20:51.181 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.190+0000 7f532effd640 1 -- 172.21.15.176:0/4206931501 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f53380b0230 con 0x7f533809ac40 2024-03-29T11:20:51.181 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.190+0000 7f532effd640 1 -- 172.21.15.176:0/4206931501 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f5328001bc0 con 0x7f533809ac40 2024-03-29T11:20:51.181 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.190+0000 7f532effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:51.181 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.190+0000 7f532effd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:20:51.181 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.190+0000 7f532effd640 10 monclient(hunting): dump: 2024-03-29T11:20:51.181 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-03-29T11:20:51.181 INFO:teuthology.orchestra.run.smithi176.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:20:51.182 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:51.182 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:51.182 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:51.182 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T11:20:51.182 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:20:51.182 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:20:51.182 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:20:51.182 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T11:20:51.182 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.190+0000 7f532effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:51.182 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.190+0000 7f532effd640 1 -- 172.21.15.176:0/4206931501 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2281286991 0 0) 0x7f5338139a00 con 0x7f53380afdc0 2024-03-29T11:20:51.182 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.190+0000 7f532effd640 1 -- 172.21.15.176:0/4206931501 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5318004eb0 con 0x7f53380afdc0 2024-03-29T11:20:51.182 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.190+0000 7f532effd640 1 -- 172.21.15.176:0/4206931501 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f5330001aa0 con 0x7f53380afdc0 2024-03-29T11:20:51.182 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.190+0000 7f532effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:51.182 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.190+0000 7f532effd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:20:51.183 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.190+0000 7f532effd640 10 monclient(hunting): dump: 2024-03-29T11:20:51.183 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-03-29T11:20:51.183 INFO:teuthology.orchestra.run.smithi176.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:20:51.183 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:51.183 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:51.183 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:51.183 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T11:20:51.183 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:20:51.183 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:20:51.183 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:20:51.183 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T11:20:51.183 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.190+0000 7f532effd640 1 -- 172.21.15.176:0/4206931501 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1707095491 0 0) 0x7f53381db610 con 0x7f5338138f60 2024-03-29T11:20:51.183 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.190+0000 7f532effd640 1 -- 172.21.15.176:0/4206931501 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5338139a00 con 0x7f5338138f60 2024-03-29T11:20:51.183 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.190+0000 7f532effd640 1 -- 172.21.15.176:0/4206931501 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f5320001860 con 0x7f5338138f60 2024-03-29T11:20:51.184 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.190+0000 7f532effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:51.184 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.190+0000 7f532effd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:20:51.184 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.190+0000 7f532effd640 10 monclient(hunting): dump: 2024-03-29T11:20:51.184 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-03-29T11:20:51.184 INFO:teuthology.orchestra.run.smithi176.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:20:51.184 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:51.184 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:51.184 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:51.184 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T11:20:51.184 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:20:51.184 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:20:51.184 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:20:51.184 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T11:20:51.184 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.190+0000 7f532effd640 1 -- 172.21.15.176:0/4206931501 <== mon.1 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2176070604 0 0) 0x7f53380b0230 con 0x7f533809ac40 2024-03-29T11:20:51.184 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.190+0000 7f532effd640 1 -- 172.21.15.176:0/4206931501 >> v1:172.21.15.114:6790/0 conn(0x7f5338138f60 legacy=0x7f53380af6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:51.185 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.190+0000 7f532effd640 1 -- 172.21.15.176:0/4206931501 >> v1:172.21.15.114:6789/0 conn(0x7f53380afdc0 legacy=0x7f53381e1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:51.185 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.190+0000 7f532effd640 10 monclient: _finish_hunting 0 2024-03-29T11:20:51.185 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.190+0000 7f532effd640 1 monclient: found mon.b 2024-03-29T11:20:51.185 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.190+0000 7f532effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:51.185 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.190+0000 7f532effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:20:51.185 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.190+0000 7f532effd640 1 -- 172.21.15.176:0/4206931501 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53380b03c0 con 0x7f533809ac40 2024-03-29T11:20:51.185 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.190+0000 7f532effd640 10 monclient: _finish_auth 0 2024-03-29T11:20:51.185 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.190+0000 7f532effd640 10 monclient: _check_auth_tickets 2024-03-29T11:20:51.185 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.190+0000 7f532effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:51.185 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.190+0000 7f533cd7c640 5 monclient: authenticate success, global_id 4141 2024-03-29T11:20:51.185 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.190+0000 7f533cd7c640 10 monclient: _renew_subs 2024-03-29T11:20:51.185 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.190+0000 7f533cd7c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:20:51.185 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.190+0000 7f533cd7c640 1 -- 172.21.15.176:0/4206931501 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53381e36e0 con 0x7f533809ac40 2024-03-29T11:20:51.185 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.190+0000 7f533cd7c640 10 client.4141.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:51.186 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.190+0000 7f533cd7c640 10 monclient: _renew_subs 2024-03-29T11:20:51.186 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.190+0000 7f533cd7c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:20:51.186 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.190+0000 7f533cd7c640 1 -- 172.21.15.176:0/4206931501 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f53381e3a50 con 0x7f533809ac40 2024-03-29T11:20:51.186 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.190+0000 7f533cd7c640 1 librados: init done 2024-03-29T11:20:51.186 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.190+0000 7f532effd640 1 -- 172.21.15.176:0/4206931501 <== mon.1 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5328004bf0 con 0x7f533809ac40 2024-03-29T11:20:51.186 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.190+0000 7f532effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:51.186 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.190+0000 7f532effd640 1 -- 172.21.15.176:0/4206931501 <== mon.1 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f5328004d60 con 0x7f533809ac40 2024-03-29T11:20:51.186 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.190+0000 7f532effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:51.186 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.190+0000 7f532effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:20:51.186 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.190+0000 7f533559a640 4 set_mon_vals no callback set 2024-03-29T11:20:51.186 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.190+0000 7f532effd640 10 monclient: dump: 2024-03-29T11:20:51.186 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-03-29T11:20:51.186 INFO:teuthology.orchestra.run.smithi176.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:20:51.186 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:51.186 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:51.187 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:51.187 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T11:20:51.187 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:20:51.187 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:20:51.187 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:20:51.187 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T11:20:51.187 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.191+0000 7f532effd640 1 -- 172.21.15.176:0/4206931501 <== mon.1 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f5328005250 con 0x7f533809ac40 2024-03-29T11:20:51.187 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.191+0000 7f532effd640 1 -- 172.21.15.176:0/4206931501 <== mon.1 v1:172.21.15.176:6789/0 8 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (unknown 1930922248 0 0) 0x7f5328001170 con 0x7f533809ac40 2024-03-29T11:20:51.187 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.191+0000 7f532effd640 10 client.4141.objecter ms_dispatch 0x7f5338000b90 osd_map(7..7 src has 1..7) v4 2024-03-29T11:20:51.187 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.191+0000 7f532effd640 3 client.4141.objecter handle_osd_map got epochs [7,7] > 0 2024-03-29T11:20:51.187 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.191+0000 7f532effd640 3 client.4141.objecter handle_osd_map decoding full epoch 7 2024-03-29T11:20:51.187 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.191+0000 7f532effd640 20 client.4141.objecter dump_active .. 0 homeless 2024-03-29T11:20:51.187 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.191+0000 7f533cd7c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:51.187 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.191+0000 7f533cd7c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:51.187 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.191+0000 7f533cd7c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:20:51.188 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.191+0000 7f533cd7c640 1 -- 172.21.15.176:0/4206931501 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f531c005080 con 0x7f533809ac40 2024-03-29T11:20:51.188 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.193+0000 7f532effd640 1 -- 172.21.15.176:0/4206931501 <== mon.1 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f5328005b80 con 0x7f533809ac40 2024-03-29T11:20:51.188 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.193+0000 7f532effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:51.188 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.193+0000 7f532effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:51.243 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.266+0000 7f533cd7c640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "ecd09e8b-8a85-4997-bdea-75d754e6ee67", "id": 5}] 2024-03-29T11:20:51.243 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.266+0000 7f533cd7c640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "ecd09e8b-8a85-4997-bdea-75d754e6ee67", "id": 5}] 2024-03-29T11:20:51.243 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.266+0000 7f533cd7c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:20:51.243 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.266+0000 7f533cd7c640 1 -- 172.21.15.176:0/4206931501 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd new", "uuid": "ecd09e8b-8a85-4997-bdea-75d754e6ee67", "id": 5} v 0) v1 -- 0x7f531c0058b0 con 0x7f533809ac40 2024-03-29T11:20:51.247 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.270+0000 7f532effd640 1 -- 172.21.15.176:0/4206931501 <== mon.1 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ecd09e8b-8a85-4997-bdea-75d754e6ee67", "id": 5}]=0 v8) v1 ==== 112+0+2 (unknown 1505400278 0 1988412176) 0x7f5328005500 con 0x7f533809ac40 2024-03-29T11:20:51.247 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.270+0000 7f532effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "ecd09e8b-8a85-4997-bdea-75d754e6ee67", "id": 5}] 2024-03-29T11:20:51.247 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.270+0000 7f532effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:51.247 INFO:teuthology.orchestra.run.smithi176.stdout:5 2024-03-29T11:20:51.248 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.271+0000 7f533cd7c640 10 librados: watch_flush enter 2024-03-29T11:20:51.248 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.271+0000 7f533cd7c640 10 librados: watch_flush exit 2024-03-29T11:20:51.248 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.271+0000 7f533cd7c640 20 client.4141.objecter shutdown clearing up homeless session... 2024-03-29T11:20:51.248 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.271+0000 7f533cd7c640 10 client.4141.objecter successfully canceled tick 2024-03-29T11:20:51.248 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.271+0000 7f533cd7c640 10 monclient: shutdown 2024-03-29T11:20:51.248 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.271+0000 7f533cd7c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:51.248 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.271+0000 7f533cd7c640 1 -- 172.21.15.176:0/4206931501 >> v1:172.21.15.176:6789/0 conn(0x7f533809ac40 legacy=0x7f53381dac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:51.249 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.271+0000 7f533cd7c640 1 -- 172.21.15.176:0/4206931501 shutdown_connections 2024-03-29T11:20:51.249 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.271+0000 7f533cd7c640 1 -- 172.21.15.176:0/4206931501 >> 172.21.15.176:0/4206931501 conn(0x7f5338066b00 msgr2=0x7f53380b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:51.249 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.271+0000 7f533cd7c640 1 -- 172.21.15.176:0/4206931501 shutdown_connections 2024-03-29T11:20:51.249 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.271+0000 7f533cd7c640 1 -- 172.21.15.176:0/4206931501 wait complete. 2024-03-29T11:20:51.249 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.271+0000 7f533cd7c640 1 librados: shutdown 2024-03-29T11:20:51.258 DEBUG:teuthology.orchestra.run.smithi176:> sudo ceph --cluster ceph osd new 2f0d9ead-f47f-4cb3-92a4-0a5ad23cd831 6 2024-03-29T11:20:51.399 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.422+0000 7f878f315640 10 monclient: get_monmap_and_config 2024-03-29T11:20:51.399 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.422+0000 7f878f315640 10 monclient: build_initial_monmap 2024-03-29T11:20:51.399 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.422+0000 7f878f315640 10 monclient: monmap: 2024-03-29T11:20:51.399 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 0 2024-03-29T11:20:51.399 INFO:teuthology.orchestra.run.smithi176.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:20:51.399 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-29T11:20:51.423077+0000 2024-03-29T11:20:51.399 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-29T11:20:51.423077+0000 2024-03-29T11:20:51.399 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:51.399 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T11:20:51.399 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:20:51.399 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T11:20:51.400 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T11:20:51.400 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T11:20:51.400 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.422+0000 7f878f315640 1 Processor -- start 2024-03-29T11:20:51.400 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.422+0000 7f878f315640 1 -- start start 2024-03-29T11:20:51.400 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.422+0000 7f878f315640 10 monclient: init 2024-03-29T11:20:51.409 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.423+0000 7f878f315640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:51.409 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.423+0000 7f878f315640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:20:51.409 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.423+0000 7f878f315640 10 monclient(hunting): picked mon.noname-b con 0x7f878809ac40 addr v1:172.21.15.114:6790/0 2024-03-29T11:20:51.409 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.423+0000 7f878f315640 10 monclient(hunting): picked mon.noname-a con 0x7f878809c870 addr v1:172.21.15.114:6789/0 2024-03-29T11:20:51.409 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.423+0000 7f878f315640 10 monclient(hunting): picked mon.noname-c con 0x7f87880b4360 addr v1:172.21.15.176:6789/0 2024-03-29T11:20:51.409 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.423+0000 7f878f315640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f878809dec0 con 0x7f878809c870 2024-03-29T11:20:51.409 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.423+0000 7f878f315640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f878809da30 con 0x7f87880b4360 2024-03-29T11:20:51.409 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.423+0000 7f878f315640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87880b39c0 con 0x7f878809ac40 2024-03-29T11:20:51.409 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.423+0000 7f878f315640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:51.409 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.423+0000 7f878f315640 10 monclient(hunting): authenticate will time out at 916.764708s 2024-03-29T11:20:51.409 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.423+0000 7f878d8b2640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f87880b4360 0x7f878814d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.176:49106/0 (socket says 172.21.15.176:49106) 2024-03-29T11:20:51.409 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.423+0000 7f878d8b2640 1 -- 172.21.15.176:0/829960916 learned_addr learned my addr 172.21.15.176:0/829960916 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-29T11:20:51.410 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.423+0000 7f877ffff640 1 -- 172.21.15.176:0/829960916 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1051128732 0 0) 0x7f878809da30 con 0x7f87880b4360 2024-03-29T11:20:51.410 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.423+0000 7f877ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:20:51.410 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.423+0000 7f877ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:51.410 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.423+0000 7f877ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:51.410 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.423+0000 7f877ffff640 10 monclient(hunting): my global_id is 4144 2024-03-29T11:20:51.410 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.423+0000 7f877ffff640 1 -- 172.21.15.176:0/829960916 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8774003520 con 0x7f87880b4360 2024-03-29T11:20:51.410 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.423+0000 7f877ffff640 1 -- 172.21.15.176:0/829960916 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3740318019 0 0) 0x7f8784001110 con 0x7f878809c870 2024-03-29T11:20:51.410 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.423+0000 7f877ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:20:51.410 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.423+0000 7f877ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:51.410 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.423+0000 7f877ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:51.410 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.423+0000 7f877ffff640 10 monclient(hunting): my global_id is 4149 2024-03-29T11:20:51.410 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.423+0000 7f877ffff640 1 -- 172.21.15.176:0/829960916 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f878809da30 con 0x7f878809c870 2024-03-29T11:20:51.410 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.423+0000 7f877ffff640 1 -- 172.21.15.176:0/829960916 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2325320107 0 0) 0x7f87880b39c0 con 0x7f878809ac40 2024-03-29T11:20:51.410 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.423+0000 7f877ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:20:51.411 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.423+0000 7f877ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:51.411 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.423+0000 7f877ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:51.411 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.423+0000 7f877ffff640 10 monclient(hunting): my global_id is 4139 2024-03-29T11:20:51.411 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.423+0000 7f877ffff640 1 -- 172.21.15.176:0/829960916 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8784001110 con 0x7f878809ac40 2024-03-29T11:20:51.411 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.424+0000 7f877ffff640 1 -- 172.21.15.176:0/829960916 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 752607771 0 0) 0x7f8774003520 con 0x7f87880b4360 2024-03-29T11:20:51.411 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.424+0000 7f877ffff640 1 -- 172.21.15.176:0/829960916 >> v1:172.21.15.114:6790/0 conn(0x7f878809ac40 legacy=0x7f87880b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:51.411 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.424+0000 7f877ffff640 1 -- 172.21.15.176:0/829960916 >> v1:172.21.15.114:6789/0 conn(0x7f878809c870 legacy=0x7f878813f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:51.411 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.424+0000 7f877ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:20:51.411 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.424+0000 7f877ffff640 1 monclient: found mon.noname-c 2024-03-29T11:20:51.411 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.424+0000 7f877ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:51.411 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.424+0000 7f877ffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.176:6789/0 2024-03-29T11:20:51.411 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.424+0000 7f877ffff640 1 -- 172.21.15.176:0/829960916 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87880b5160 con 0x7f87880b4360 2024-03-29T11:20:51.411 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.424+0000 7f877ffff640 10 monclient: _finish_auth 0 2024-03-29T11:20:51.411 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.424+0000 7f877ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:20:51.412 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.424+0000 7f877ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:51.412 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.424+0000 7f877ffff640 1 -- 172.21.15.176:0/829960916 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f8780001a10 con 0x7f87880b4360 2024-03-29T11:20:51.412 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.424+0000 7f878f315640 5 monclient: authenticate success, global_id 4144 2024-03-29T11:20:51.412 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.424+0000 7f878f315640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:51.412 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.424+0000 7f877ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:51.412 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.424+0000 7f877ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:20:51.412 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.424+0000 7f877ffff640 10 monclient: dump: 2024-03-29T11:20:51.412 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-03-29T11:20:51.412 INFO:teuthology.orchestra.run.smithi176.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:20:51.412 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:51.412 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:51.412 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:51.412 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T11:20:51.412 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:20:51.412 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:20:51.413 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:20:51.413 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T11:20:51.413 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.424+0000 7f878f315640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:51.413 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.424+0000 7f877ffff640 1 -- 172.21.15.176:0/829960916 <== mon.1 v1:172.21.15.176:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8780001bf0 con 0x7f87880b4360 2024-03-29T11:20:51.413 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.424+0000 7f877ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:51.413 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.424+0000 7f877ffff640 1 -- 172.21.15.176:0/829960916 <== mon.1 v1:172.21.15.176:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f8780002000 con 0x7f87880b4360 2024-03-29T11:20:51.413 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.424+0000 7f877ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:51.413 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.424+0000 7f877ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:20:51.413 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.424+0000 7f877ffff640 10 monclient: dump: 2024-03-29T11:20:51.413 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-03-29T11:20:51.413 INFO:teuthology.orchestra.run.smithi176.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:20:51.413 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:51.413 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:51.413 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:51.413 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T11:20:51.414 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:20:51.414 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:20:51.414 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:20:51.414 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T11:20:51.414 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.424+0000 7f878f315640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:51.414 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.424+0000 7f878f315640 4 set_mon_vals no callback set 2024-03-29T11:20:51.414 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.424+0000 7f878f315640 10 monclient: shutdown 2024-03-29T11:20:51.414 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.424+0000 7f878f315640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:51.414 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.424+0000 7f878f315640 1 -- 172.21.15.176:0/829960916 >> v1:172.21.15.176:6789/0 conn(0x7f87880b4360 legacy=0x7f878814d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:51.414 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.425+0000 7f878f315640 1 -- 172.21.15.176:0/829960916 shutdown_connections 2024-03-29T11:20:51.414 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.425+0000 7f878f315640 1 -- 172.21.15.176:0/829960916 >> 172.21.15.176:0/829960916 conn(0x7f8788066b00 msgr2=0x7f8788066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:51.414 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.425+0000 7f878f315640 1 -- 172.21.15.176:0/829960916 shutdown_connections 2024-03-29T11:20:51.415 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.425+0000 7f878f315640 1 -- 172.21.15.176:0/829960916 wait complete. 2024-03-29T11:20:51.415 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.425+0000 7f878f315640 10 monclient: build_initial_monmap 2024-03-29T11:20:51.415 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.425+0000 7f878f315640 10 monclient: monmap: 2024-03-29T11:20:51.415 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 0 2024-03-29T11:20:51.415 INFO:teuthology.orchestra.run.smithi176.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:51.415 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 0.000000 2024-03-29T11:20:51.415 INFO:teuthology.orchestra.run.smithi176.stderr:created 0.000000 2024-03-29T11:20:51.415 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:51.415 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T11:20:51.415 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:20:51.415 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T11:20:51.415 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T11:20:51.415 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T11:20:51.415 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.425+0000 7f878f315640 1 librados: starting msgr at 2024-03-29T11:20:51.416 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.425+0000 7f878f315640 1 librados: starting objecter 2024-03-29T11:20:51.416 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.426+0000 7f878f315640 1 Processor -- start 2024-03-29T11:20:51.416 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.426+0000 7f878f315640 1 -- start start 2024-03-29T11:20:51.416 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.426+0000 7f878f315640 1 librados: setting wanted keys 2024-03-29T11:20:51.416 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.426+0000 7f878f315640 1 librados: calling monclient init 2024-03-29T11:20:51.416 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.426+0000 7f878f315640 10 monclient: init 2024-03-29T11:20:51.416 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.426+0000 7f878f315640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:51.416 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.426+0000 7f878f315640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:20:51.416 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.426+0000 7f878f315640 10 monclient(hunting): picked mon.noname-c con 0x7f878809ac40 addr v1:172.21.15.114:6790/0 2024-03-29T11:20:51.416 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.426+0000 7f878f315640 10 monclient(hunting): picked mon.noname-a con 0x7f87881d7cd0 addr v1:172.21.15.114:6789/0 2024-03-29T11:20:51.416 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.426+0000 7f878f315640 10 monclient(hunting): picked mon.noname-b con 0x7f87880ad1e0 addr v1:172.21.15.176:6789/0 2024-03-29T11:20:51.416 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.426+0000 7f878f315640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87881d7130 con 0x7f87881d7cd0 2024-03-29T11:20:51.416 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.426+0000 7f878f315640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8788139b10 con 0x7f87880ad1e0 2024-03-29T11:20:51.416 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.426+0000 7f878f315640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f878813ffa0 con 0x7f878809ac40 2024-03-29T11:20:51.417 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.426+0000 7f878f315640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:51.417 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.426+0000 7f878f315640 10 monclient(hunting): authenticate will time out at 916.767813s 2024-03-29T11:20:51.417 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.426+0000 7f878d8b2640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f87880ad1e0 0x7f8788139400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.176:49116/0 (socket says 172.21.15.176:49116) 2024-03-29T11:20:51.417 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.426+0000 7f878d8b2640 1 -- 172.21.15.176:0/3838437368 learned_addr learned my addr 172.21.15.176:0/3838437368 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-29T11:20:51.417 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.426+0000 7f877dffb640 10 client.?.objecter ms_handle_connect 0x7f87880ad1e0 2024-03-29T11:20:51.417 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.426+0000 7f877dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:51.417 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.426+0000 7f877dffb640 10 client.?.objecter ms_handle_connect 0x7f87881d7cd0 2024-03-29T11:20:51.417 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.426+0000 7f877dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:51.417 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.426+0000 7f877dffb640 10 client.?.objecter ms_handle_connect 0x7f878809ac40 2024-03-29T11:20:51.417 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.426+0000 7f877dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:51.417 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.426+0000 7f877dffb640 1 -- 172.21.15.176:0/3838437368 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3839496121 0 0) 0x7f8788139b10 con 0x7f87880ad1e0 2024-03-29T11:20:51.417 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.426+0000 7f877dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:20:51.417 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.426+0000 7f877dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:51.417 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.426+0000 7f877dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:51.418 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.426+0000 7f877dffb640 10 monclient(hunting): my global_id is 4147 2024-03-29T11:20:51.418 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.426+0000 7f877dffb640 1 -- 172.21.15.176:0/3838437368 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f876c004eb0 con 0x7f87880ad1e0 2024-03-29T11:20:51.418 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.427+0000 7f877dffb640 1 -- 172.21.15.176:0/3838437368 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 217628457 0 0) 0x7f87881d7130 con 0x7f87881d7cd0 2024-03-29T11:20:51.418 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.427+0000 7f877dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:20:51.418 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.427+0000 7f877dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:51.418 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.427+0000 7f877dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:51.418 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.427+0000 7f877dffb640 10 monclient(hunting): my global_id is 4152 2024-03-29T11:20:51.418 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.427+0000 7f877dffb640 1 -- 172.21.15.176:0/3838437368 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8788139b10 con 0x7f87881d7cd0 2024-03-29T11:20:51.418 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.427+0000 7f877dffb640 1 -- 172.21.15.176:0/3838437368 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2357474578 0 0) 0x7f878813ffa0 con 0x7f878809ac40 2024-03-29T11:20:51.418 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.427+0000 7f877dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:20:51.418 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.427+0000 7f877dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:51.418 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.427+0000 7f877dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:51.418 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.427+0000 7f877dffb640 10 monclient(hunting): my global_id is 4142 2024-03-29T11:20:51.419 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.427+0000 7f877dffb640 1 -- 172.21.15.176:0/3838437368 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87881d7130 con 0x7f878809ac40 2024-03-29T11:20:51.419 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.427+0000 7f877dffb640 1 -- 172.21.15.176:0/3838437368 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1254765767 0 0) 0x7f876c004eb0 con 0x7f87880ad1e0 2024-03-29T11:20:51.419 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.427+0000 7f877dffb640 1 -- 172.21.15.176:0/3838437368 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f878813ffa0 con 0x7f87880ad1e0 2024-03-29T11:20:51.419 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.427+0000 7f877dffb640 1 -- 172.21.15.176:0/3838437368 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f8780001f40 con 0x7f87880ad1e0 2024-03-29T11:20:51.419 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.427+0000 7f877dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:51.419 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.427+0000 7f877dffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:20:51.419 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.427+0000 7f877dffb640 10 monclient(hunting): dump: 2024-03-29T11:20:51.419 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-03-29T11:20:51.419 INFO:teuthology.orchestra.run.smithi176.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:20:51.419 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:51.419 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:51.419 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:51.419 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T11:20:51.419 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:20:51.419 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:20:51.420 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:20:51.420 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T11:20:51.420 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.427+0000 7f877dffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:51.420 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.427+0000 7f877dffb640 1 -- 172.21.15.176:0/3838437368 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 336670182 0 0) 0x7f8788139b10 con 0x7f87881d7cd0 2024-03-29T11:20:51.420 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.427+0000 7f877dffb640 1 -- 172.21.15.176:0/3838437368 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f876c004eb0 con 0x7f87881d7cd0 2024-03-29T11:20:51.420 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.427+0000 7f877dffb640 1 -- 172.21.15.176:0/3838437368 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f87840018e0 con 0x7f87881d7cd0 2024-03-29T11:20:51.420 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.427+0000 7f877dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:51.420 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.427+0000 7f877dffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:20:51.420 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.427+0000 7f877dffb640 10 monclient(hunting): dump: 2024-03-29T11:20:51.420 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-03-29T11:20:51.420 INFO:teuthology.orchestra.run.smithi176.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:20:51.420 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:51.420 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:51.420 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:51.420 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T11:20:51.420 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:20:51.420 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:20:51.421 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:20:51.421 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T11:20:51.421 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.427+0000 7f877dffb640 1 -- 172.21.15.176:0/3838437368 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2506712279 0 0) 0x7f87881d7130 con 0x7f878809ac40 2024-03-29T11:20:51.421 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.427+0000 7f877dffb640 1 -- 172.21.15.176:0/3838437368 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8788139b10 con 0x7f878809ac40 2024-03-29T11:20:51.421 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.427+0000 7f877dffb640 1 -- 172.21.15.176:0/3838437368 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f87780017f0 con 0x7f878809ac40 2024-03-29T11:20:51.421 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.427+0000 7f877dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:51.421 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.427+0000 7f877dffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:20:51.421 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.427+0000 7f877dffb640 10 monclient(hunting): dump: 2024-03-29T11:20:51.421 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-03-29T11:20:51.421 INFO:teuthology.orchestra.run.smithi176.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:20:51.421 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:51.421 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:51.421 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:51.421 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T11:20:51.422 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:20:51.422 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:20:51.422 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:20:51.422 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T11:20:51.422 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.427+0000 7f877dffb640 1 -- 172.21.15.176:0/3838437368 <== mon.1 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 895153473 0 0) 0x7f878813ffa0 con 0x7f87880ad1e0 2024-03-29T11:20:51.422 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.427+0000 7f877dffb640 1 -- 172.21.15.176:0/3838437368 >> v1:172.21.15.114:6790/0 conn(0x7f878809ac40 legacy=0x7f87881d75c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:51.422 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.427+0000 7f877dffb640 1 -- 172.21.15.176:0/3838437368 >> v1:172.21.15.114:6789/0 conn(0x7f87881d7cd0 legacy=0x7f8788138cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:51.422 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.427+0000 7f877dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:20:51.422 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.427+0000 7f877dffb640 1 monclient: found mon.b 2024-03-29T11:20:51.422 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.427+0000 7f877dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:51.422 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.427+0000 7f877dffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:20:51.422 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.427+0000 7f877dffb640 1 -- 172.21.15.176:0/3838437368 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87880ae520 con 0x7f87880ad1e0 2024-03-29T11:20:51.422 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.427+0000 7f877dffb640 10 monclient: _finish_auth 0 2024-03-29T11:20:51.422 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.427+0000 7f877dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:20:51.423 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.427+0000 7f877dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:51.423 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.427+0000 7f878f315640 5 monclient: authenticate success, global_id 4147 2024-03-29T11:20:51.423 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.427+0000 7f878f315640 10 monclient: _renew_subs 2024-03-29T11:20:51.423 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.427+0000 7f878f315640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:20:51.423 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.427+0000 7f878f315640 1 -- 172.21.15.176:0/3838437368 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87880ae800 con 0x7f87880ad1e0 2024-03-29T11:20:51.423 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.427+0000 7f877dffb640 1 -- 172.21.15.176:0/3838437368 <== mon.1 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8780004e10 con 0x7f87880ad1e0 2024-03-29T11:20:51.423 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.427+0000 7f877dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:51.423 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.427+0000 7f877dffb640 1 -- 172.21.15.176:0/3838437368 <== mon.1 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f8780005120 con 0x7f87880ad1e0 2024-03-29T11:20:51.423 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.427+0000 7f877dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:51.423 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.427+0000 7f877dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:20:51.423 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.427+0000 7f877dffb640 10 monclient: dump: 2024-03-29T11:20:51.423 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-03-29T11:20:51.423 INFO:teuthology.orchestra.run.smithi176.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:20:51.423 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:51.423 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:51.424 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:51.424 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T11:20:51.424 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:20:51.424 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:20:51.424 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:20:51.424 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T11:20:51.424 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.427+0000 7f877f7fe640 4 set_mon_vals no callback set 2024-03-29T11:20:51.424 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.427+0000 7f878f315640 10 client.4147.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:51.424 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.427+0000 7f878f315640 10 monclient: _renew_subs 2024-03-29T11:20:51.424 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.427+0000 7f878f315640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:20:51.424 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.427+0000 7f877dffb640 1 -- 172.21.15.176:0/3838437368 <== mon.1 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f8780005690 con 0x7f87880ad1e0 2024-03-29T11:20:51.424 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.427+0000 7f878f315640 1 -- 172.21.15.176:0/3838437368 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8788142b70 con 0x7f87880ad1e0 2024-03-29T11:20:51.424 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.427+0000 7f878f315640 1 librados: init done 2024-03-29T11:20:51.424 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.427+0000 7f877dffb640 1 -- 172.21.15.176:0/3838437368 <== mon.1 v1:172.21.15.176:6789/0 8 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (unknown 2291270142 0 0) 0x7f8780005f30 con 0x7f87880ad1e0 2024-03-29T11:20:51.424 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.427+0000 7f877dffb640 10 client.4147.objecter ms_dispatch 0x7f8788000b90 osd_map(8..8 src has 1..8) v4 2024-03-29T11:20:51.425 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.427+0000 7f877dffb640 3 client.4147.objecter handle_osd_map got epochs [8,8] > 0 2024-03-29T11:20:51.425 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.427+0000 7f877dffb640 3 client.4147.objecter handle_osd_map decoding full epoch 8 2024-03-29T11:20:51.425 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.427+0000 7f877dffb640 20 client.4147.objecter dump_active .. 0 homeless 2024-03-29T11:20:51.425 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.428+0000 7f878f315640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:51.425 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.428+0000 7f878f315640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:51.425 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.428+0000 7f878f315640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:20:51.425 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.428+0000 7f878f315640 1 -- 172.21.15.176:0/3838437368 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f878809b020 con 0x7f87880ad1e0 2024-03-29T11:20:51.425 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.430+0000 7f877dffb640 1 -- 172.21.15.176:0/3838437368 <== mon.1 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f8780006170 con 0x7f87880ad1e0 2024-03-29T11:20:51.425 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.430+0000 7f877dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:51.425 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.430+0000 7f877dffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:51.426 INFO:tasks.ceph.mgr.y.smithi114.stderr:2024-03-29T11:20:51.439+0000 7ffbece52040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T11:20:51.479 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.502+0000 7f878f315640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "2f0d9ead-f47f-4cb3-92a4-0a5ad23cd831", "id": 6}] 2024-03-29T11:20:51.479 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.502+0000 7f878f315640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "2f0d9ead-f47f-4cb3-92a4-0a5ad23cd831", "id": 6}] 2024-03-29T11:20:51.479 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.502+0000 7f878f315640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:20:51.479 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.502+0000 7f878f315640 1 -- 172.21.15.176:0/3838437368 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd new", "uuid": "2f0d9ead-f47f-4cb3-92a4-0a5ad23cd831", "id": 6} v 0) v1 -- 0x7f8788142ec0 con 0x7f87880ad1e0 2024-03-29T11:20:51.483 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.506+0000 7f877dffb640 1 -- 172.21.15.176:0/3838437368 <== mon.1 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "2f0d9ead-f47f-4cb3-92a4-0a5ad23cd831", "id": 6}]=0 v9) v1 ==== 112+0+2 (unknown 3605336940 0 1113811849) 0x7f87800063e0 con 0x7f87880ad1e0 2024-03-29T11:20:51.483 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.506+0000 7f877dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "2f0d9ead-f47f-4cb3-92a4-0a5ad23cd831", "id": 6}] 2024-03-29T11:20:51.483 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.506+0000 7f877dffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:51.483 INFO:teuthology.orchestra.run.smithi176.stdout:6 2024-03-29T11:20:51.484 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.507+0000 7f878f315640 10 librados: watch_flush enter 2024-03-29T11:20:51.485 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.507+0000 7f878f315640 10 librados: watch_flush exit 2024-03-29T11:20:51.485 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.507+0000 7f878f315640 20 client.4147.objecter shutdown clearing up homeless session... 2024-03-29T11:20:51.485 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.507+0000 7f878f315640 10 client.4147.objecter successfully canceled tick 2024-03-29T11:20:51.485 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.508+0000 7f878f315640 10 monclient: shutdown 2024-03-29T11:20:51.485 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.508+0000 7f878f315640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:51.485 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.508+0000 7f878f315640 1 -- 172.21.15.176:0/3838437368 >> v1:172.21.15.176:6789/0 conn(0x7f87880ad1e0 legacy=0x7f8788139400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:51.485 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.508+0000 7f878f315640 1 -- 172.21.15.176:0/3838437368 shutdown_connections 2024-03-29T11:20:51.485 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.508+0000 7f878f315640 1 -- 172.21.15.176:0/3838437368 >> 172.21.15.176:0/3838437368 conn(0x7f8788066b00 msgr2=0x7f87880b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:51.485 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.508+0000 7f878f315640 1 -- 172.21.15.176:0/3838437368 shutdown_connections 2024-03-29T11:20:51.485 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.508+0000 7f878f315640 1 -- 172.21.15.176:0/3838437368 wait complete. 2024-03-29T11:20:51.485 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.508+0000 7f878f315640 1 librados: shutdown 2024-03-29T11:20:51.495 DEBUG:teuthology.orchestra.run.smithi176:> sudo ceph --cluster ceph osd new fdb6a25f-6bd8-4c65-9b9b-b460ba7673a1 7 2024-03-29T11:20:51.497 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-03-29T11:20:51.520+0000 7fedaa420040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T11:20:51.552 INFO:tasks.ceph.mgr.y.smithi114.stderr:2024-03-29T11:20:51.575+0000 7ffbece52040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T11:20:51.594 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.618+0000 7f9aead54640 10 monclient: get_monmap_and_config 2024-03-29T11:20:51.595 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.618+0000 7f9aead54640 10 monclient: build_initial_monmap 2024-03-29T11:20:51.595 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.618+0000 7f9aead54640 10 monclient: monmap: 2024-03-29T11:20:51.595 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 0 2024-03-29T11:20:51.595 INFO:teuthology.orchestra.run.smithi176.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:20:51.595 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-29T11:20:51.619160+0000 2024-03-29T11:20:51.595 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-29T11:20:51.619160+0000 2024-03-29T11:20:51.595 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:51.595 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T11:20:51.595 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:20:51.595 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T11:20:51.595 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T11:20:51.595 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T11:20:51.604 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.618+0000 7f9aead54640 1 Processor -- start 2024-03-29T11:20:51.604 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.618+0000 7f9aead54640 1 -- start start 2024-03-29T11:20:51.604 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.618+0000 7f9aead54640 10 monclient: init 2024-03-29T11:20:51.604 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.619+0000 7f9aead54640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:51.604 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.619+0000 7f9aead54640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:20:51.604 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.619+0000 7f9aead54640 10 monclient(hunting): picked mon.noname-b con 0x7f9ae409ac40 addr v1:172.21.15.114:6790/0 2024-03-29T11:20:51.604 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.619+0000 7f9aead54640 10 monclient(hunting): picked mon.noname-a con 0x7f9ae409c870 addr v1:172.21.15.114:6789/0 2024-03-29T11:20:51.604 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.619+0000 7f9aead54640 10 monclient(hunting): picked mon.noname-c con 0x7f9ae40b4360 addr v1:172.21.15.176:6789/0 2024-03-29T11:20:51.605 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.619+0000 7f9aead54640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ae409dec0 con 0x7f9ae409c870 2024-03-29T11:20:51.605 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.619+0000 7f9aead54640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ae409da30 con 0x7f9ae40b4360 2024-03-29T11:20:51.605 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.619+0000 7f9aead54640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ae40b39c0 con 0x7f9ae409ac40 2024-03-29T11:20:51.605 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.619+0000 7f9aead54640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:51.605 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.619+0000 7f9aead54640 10 monclient(hunting): authenticate will time out at 916.960657s 2024-03-29T11:20:51.605 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.619+0000 7f9ae92f1640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f9ae40b4360 0x7f9ae414d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.176:49118/0 (socket says 172.21.15.176:49118) 2024-03-29T11:20:51.605 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.619+0000 7f9ae92f1640 1 -- 172.21.15.176:0/3137902399 learned_addr learned my addr 172.21.15.176:0/3137902399 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-29T11:20:51.605 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.619+0000 7f9ae37fe640 1 -- 172.21.15.176:0/3137902399 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3758918086 0 0) 0x7f9ae409da30 con 0x7f9ae40b4360 2024-03-29T11:20:51.605 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.619+0000 7f9ae37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:20:51.605 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.619+0000 7f9ae37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:51.605 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.619+0000 7f9ae37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:51.605 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.619+0000 7f9ae37fe640 10 monclient(hunting): my global_id is 4150 2024-03-29T11:20:51.605 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.619+0000 7f9ae37fe640 1 -- 172.21.15.176:0/3137902399 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ad4003520 con 0x7f9ae40b4360 2024-03-29T11:20:51.605 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.619+0000 7f9ae37fe640 1 -- 172.21.15.176:0/3137902399 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 780593325 0 0) 0x7f9ad0001110 con 0x7f9ae409c870 2024-03-29T11:20:51.606 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.619+0000 7f9ae37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:20:51.606 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.619+0000 7f9ae37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:51.606 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.619+0000 7f9ae37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:51.606 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.619+0000 7f9ae37fe640 10 monclient(hunting): my global_id is 4155 2024-03-29T11:20:51.606 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.619+0000 7f9ae37fe640 1 -- 172.21.15.176:0/3137902399 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ae409da30 con 0x7f9ae409c870 2024-03-29T11:20:51.606 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.619+0000 7f9ae37fe640 1 -- 172.21.15.176:0/3137902399 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3290841346 0 0) 0x7f9ae40b39c0 con 0x7f9ae409ac40 2024-03-29T11:20:51.606 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.619+0000 7f9ae37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:20:51.606 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.619+0000 7f9ae37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:51.606 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.619+0000 7f9ae37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:51.606 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.619+0000 7f9ae37fe640 10 monclient(hunting): my global_id is 4145 2024-03-29T11:20:51.606 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.619+0000 7f9ae37fe640 1 -- 172.21.15.176:0/3137902399 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ad0001110 con 0x7f9ae409ac40 2024-03-29T11:20:51.606 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.620+0000 7f9ae37fe640 1 -- 172.21.15.176:0/3137902399 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2344122361 0 0) 0x7f9ad4003520 con 0x7f9ae40b4360 2024-03-29T11:20:51.606 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.620+0000 7f9ae37fe640 1 -- 172.21.15.176:0/3137902399 >> v1:172.21.15.114:6790/0 conn(0x7f9ae409ac40 legacy=0x7f9ae40b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:51.606 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.620+0000 7f9ae37fe640 1 -- 172.21.15.176:0/3137902399 >> v1:172.21.15.114:6789/0 conn(0x7f9ae409c870 legacy=0x7f9ae413f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:51.607 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.620+0000 7f9ae37fe640 10 monclient: _finish_hunting 0 2024-03-29T11:20:51.607 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.620+0000 7f9ae37fe640 1 monclient: found mon.noname-c 2024-03-29T11:20:51.607 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.620+0000 7f9ae37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:51.607 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.620+0000 7f9ae37fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.176:6789/0 2024-03-29T11:20:51.607 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.620+0000 7f9ae37fe640 1 -- 172.21.15.176:0/3137902399 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ae40b5160 con 0x7f9ae40b4360 2024-03-29T11:20:51.607 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.620+0000 7f9ae37fe640 10 monclient: _finish_auth 0 2024-03-29T11:20:51.607 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.620+0000 7f9ae37fe640 10 monclient: _check_auth_tickets 2024-03-29T11:20:51.607 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.620+0000 7f9ae37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:51.607 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.620+0000 7f9aead54640 5 monclient: authenticate success, global_id 4150 2024-03-29T11:20:51.607 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.620+0000 7f9aead54640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:51.607 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.620+0000 7f9ae37fe640 1 -- 172.21.15.176:0/3137902399 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f9adc001a10 con 0x7f9ae40b4360 2024-03-29T11:20:51.607 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.620+0000 7f9ae37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:51.607 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.620+0000 7f9ae37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:20:51.607 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.620+0000 7f9ae37fe640 10 monclient: dump: 2024-03-29T11:20:51.608 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-03-29T11:20:51.608 INFO:teuthology.orchestra.run.smithi176.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:20:51.608 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:51.608 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:51.608 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:51.608 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T11:20:51.608 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:20:51.608 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:20:51.608 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:20:51.608 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T11:20:51.608 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.620+0000 7f9aead54640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:51.608 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.620+0000 7f9ae37fe640 1 -- 172.21.15.176:0/3137902399 <== mon.1 v1:172.21.15.176:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9adc001bf0 con 0x7f9ae40b4360 2024-03-29T11:20:51.608 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.620+0000 7f9ae37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:51.608 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.620+0000 7f9ae37fe640 1 -- 172.21.15.176:0/3137902399 <== mon.1 v1:172.21.15.176:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f9adc002000 con 0x7f9ae40b4360 2024-03-29T11:20:51.608 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.620+0000 7f9aead54640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:51.608 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.620+0000 7f9aead54640 4 set_mon_vals no callback set 2024-03-29T11:20:51.609 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.620+0000 7f9ae37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:51.609 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.620+0000 7f9ae37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:20:51.609 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.620+0000 7f9ae37fe640 10 monclient: dump: 2024-03-29T11:20:51.609 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-03-29T11:20:51.609 INFO:teuthology.orchestra.run.smithi176.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:20:51.609 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:51.609 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:51.609 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:51.609 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T11:20:51.609 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:20:51.609 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:20:51.609 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:20:51.609 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T11:20:51.609 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.620+0000 7f9aead54640 10 monclient: shutdown 2024-03-29T11:20:51.610 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.620+0000 7f9aead54640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:51.610 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.620+0000 7f9aead54640 1 -- 172.21.15.176:0/3137902399 >> v1:172.21.15.176:6789/0 conn(0x7f9ae40b4360 legacy=0x7f9ae414d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:51.610 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.620+0000 7f9aead54640 1 -- 172.21.15.176:0/3137902399 shutdown_connections 2024-03-29T11:20:51.610 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.620+0000 7f9aead54640 1 -- 172.21.15.176:0/3137902399 >> 172.21.15.176:0/3137902399 conn(0x7f9ae4066b00 msgr2=0x7f9ae4066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:51.610 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.620+0000 7f9aead54640 1 -- 172.21.15.176:0/3137902399 shutdown_connections 2024-03-29T11:20:51.610 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.620+0000 7f9aead54640 1 -- 172.21.15.176:0/3137902399 wait complete. 2024-03-29T11:20:51.610 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.621+0000 7f9aead54640 10 monclient: build_initial_monmap 2024-03-29T11:20:51.610 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.621+0000 7f9aead54640 10 monclient: monmap: 2024-03-29T11:20:51.610 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 0 2024-03-29T11:20:51.610 INFO:teuthology.orchestra.run.smithi176.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:51.610 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 0.000000 2024-03-29T11:20:51.610 INFO:teuthology.orchestra.run.smithi176.stderr:created 0.000000 2024-03-29T11:20:51.610 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:51.610 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T11:20:51.611 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:20:51.611 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T11:20:51.611 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T11:20:51.611 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T11:20:51.611 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.621+0000 7f9aead54640 1 librados: starting msgr at 2024-03-29T11:20:51.611 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.621+0000 7f9aead54640 1 librados: starting objecter 2024-03-29T11:20:51.611 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.621+0000 7f9aead54640 1 Processor -- start 2024-03-29T11:20:51.611 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.621+0000 7f9aead54640 1 -- start start 2024-03-29T11:20:51.611 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.621+0000 7f9aead54640 1 librados: setting wanted keys 2024-03-29T11:20:51.611 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.621+0000 7f9aead54640 1 librados: calling monclient init 2024-03-29T11:20:51.611 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.621+0000 7f9aead54640 10 monclient: init 2024-03-29T11:20:51.611 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.622+0000 7f9aead54640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:51.611 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.622+0000 7f9aead54640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:20:51.611 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.622+0000 7f9aead54640 10 monclient(hunting): picked mon.noname-c con 0x7f9ae409ac40 addr v1:172.21.15.114:6790/0 2024-03-29T11:20:51.611 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.622+0000 7f9aead54640 10 monclient(hunting): picked mon.noname-b con 0x7f9ae41d7cd0 addr v1:172.21.15.176:6789/0 2024-03-29T11:20:51.612 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.622+0000 7f9aead54640 10 monclient(hunting): picked mon.noname-a con 0x7f9ae40ad1e0 addr v1:172.21.15.114:6789/0 2024-03-29T11:20:51.612 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.622+0000 7f9aead54640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ae41d7130 con 0x7f9ae40ad1e0 2024-03-29T11:20:51.612 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.622+0000 7f9aead54640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ae4139b10 con 0x7f9ae41d7cd0 2024-03-29T11:20:51.612 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.622+0000 7f9aead54640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ae413ffa0 con 0x7f9ae409ac40 2024-03-29T11:20:51.612 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.622+0000 7f9aead54640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:51.612 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.622+0000 7f9aead54640 10 monclient(hunting): authenticate will time out at 916.963659s 2024-03-29T11:20:51.612 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.622+0000 7f9ae3fff640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f9ae41d7cd0 0x7f9ae4138cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.176:49120/0 (socket says 172.21.15.176:49120) 2024-03-29T11:20:51.612 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.622+0000 7f9ae3fff640 1 -- 172.21.15.176:0/3558186036 learned_addr learned my addr 172.21.15.176:0/3558186036 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-03-29T11:20:51.612 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.622+0000 7f9ae17fa640 10 client.?.objecter ms_handle_connect 0x7f9ae41d7cd0 2024-03-29T11:20:51.612 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.622+0000 7f9ae17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:51.612 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.622+0000 7f9ae17fa640 10 client.?.objecter ms_handle_connect 0x7f9ae40ad1e0 2024-03-29T11:20:51.612 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.622+0000 7f9ae17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:51.612 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.622+0000 7f9ae17fa640 10 client.?.objecter ms_handle_connect 0x7f9ae409ac40 2024-03-29T11:20:51.612 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.622+0000 7f9ae17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:51.613 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.622+0000 7f9ae17fa640 1 -- 172.21.15.176:0/3558186036 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2063502135 0 0) 0x7f9ae4139b10 con 0x7f9ae41d7cd0 2024-03-29T11:20:51.613 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.622+0000 7f9ae17fa640 10 monclient(hunting): _negotiate 2024-03-29T11:20:51.613 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.622+0000 7f9ae17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:51.613 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.622+0000 7f9ae17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:51.613 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.622+0000 7f9ae17fa640 10 monclient(hunting): my global_id is 4153 2024-03-29T11:20:51.613 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.622+0000 7f9ae17fa640 1 -- 172.21.15.176:0/3558186036 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ac8004eb0 con 0x7f9ae41d7cd0 2024-03-29T11:20:51.613 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.622+0000 7f9ae17fa640 1 -- 172.21.15.176:0/3558186036 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2400219552 0 0) 0x7f9ae413ffa0 con 0x7f9ae409ac40 2024-03-29T11:20:51.613 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.622+0000 7f9ae17fa640 10 monclient(hunting): _negotiate 2024-03-29T11:20:51.613 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.622+0000 7f9ae17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:51.613 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.622+0000 7f9ae17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:51.613 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.622+0000 7f9ae17fa640 10 monclient(hunting): my global_id is 4148 2024-03-29T11:20:51.613 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.622+0000 7f9ae17fa640 1 -- 172.21.15.176:0/3558186036 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ae4139b10 con 0x7f9ae409ac40 2024-03-29T11:20:51.613 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.622+0000 7f9ae17fa640 1 -- 172.21.15.176:0/3558186036 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1133232649 0 0) 0x7f9ae41d7130 con 0x7f9ae40ad1e0 2024-03-29T11:20:51.613 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.622+0000 7f9ae17fa640 10 monclient(hunting): _negotiate 2024-03-29T11:20:51.614 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.622+0000 7f9ae17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:51.614 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.622+0000 7f9ae17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:51.614 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.622+0000 7f9ae17fa640 10 monclient(hunting): my global_id is 4158 2024-03-29T11:20:51.614 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.622+0000 7f9ae17fa640 1 -- 172.21.15.176:0/3558186036 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ae413ffa0 con 0x7f9ae40ad1e0 2024-03-29T11:20:51.614 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.623+0000 7f9ae17fa640 1 -- 172.21.15.176:0/3558186036 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3641635522 0 0) 0x7f9ac8004eb0 con 0x7f9ae41d7cd0 2024-03-29T11:20:51.614 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.623+0000 7f9ae17fa640 1 -- 172.21.15.176:0/3558186036 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9ae41d7130 con 0x7f9ae41d7cd0 2024-03-29T11:20:51.614 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.623+0000 7f9ae17fa640 1 -- 172.21.15.176:0/3558186036 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f9ad00019a0 con 0x7f9ae41d7cd0 2024-03-29T11:20:51.614 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.623+0000 7f9ae17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:51.614 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.623+0000 7f9ae17fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:20:51.614 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.623+0000 7f9ae17fa640 10 monclient(hunting): dump: 2024-03-29T11:20:51.614 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-03-29T11:20:51.614 INFO:teuthology.orchestra.run.smithi176.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:20:51.614 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:51.615 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:51.615 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:51.615 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T11:20:51.615 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:20:51.615 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:20:51.615 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:20:51.615 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T11:20:51.615 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.623+0000 7f9ae17fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:51.615 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.623+0000 7f9ae17fa640 1 -- 172.21.15.176:0/3558186036 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2664251828 0 0) 0x7f9ae413ffa0 con 0x7f9ae40ad1e0 2024-03-29T11:20:51.615 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.623+0000 7f9ae17fa640 1 -- 172.21.15.176:0/3558186036 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9ac8004eb0 con 0x7f9ae40ad1e0 2024-03-29T11:20:51.615 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.623+0000 7f9ae17fa640 1 -- 172.21.15.176:0/3558186036 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1264584444 0 0) 0x7f9ae4139b10 con 0x7f9ae409ac40 2024-03-29T11:20:51.615 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.623+0000 7f9ae17fa640 1 -- 172.21.15.176:0/3558186036 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9ae413ffa0 con 0x7f9ae409ac40 2024-03-29T11:20:51.615 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.623+0000 7f9ae17fa640 1 -- 172.21.15.176:0/3558186036 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f9adc001d60 con 0x7f9ae40ad1e0 2024-03-29T11:20:51.615 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.623+0000 7f9ae17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:51.616 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.623+0000 7f9ae17fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:20:51.616 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.623+0000 7f9ae17fa640 10 monclient(hunting): dump: 2024-03-29T11:20:51.616 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-03-29T11:20:51.616 INFO:teuthology.orchestra.run.smithi176.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:20:51.616 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:51.616 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:51.616 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:51.616 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T11:20:51.616 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:20:51.616 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:20:51.616 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:20:51.616 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T11:20:51.616 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.623+0000 7f9ae17fa640 1 -- 172.21.15.176:0/3558186036 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f9ad8001760 con 0x7f9ae409ac40 2024-03-29T11:20:51.616 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.623+0000 7f9ae17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:51.617 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.623+0000 7f9ae17fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:20:51.617 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.623+0000 7f9ae17fa640 10 monclient(hunting): dump: 2024-03-29T11:20:51.617 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-03-29T11:20:51.617 INFO:teuthology.orchestra.run.smithi176.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:20:51.617 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:51.617 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:51.617 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:51.617 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T11:20:51.617 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:20:51.617 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:20:51.617 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:20:51.617 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T11:20:51.617 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.623+0000 7f9ae17fa640 1 -- 172.21.15.176:0/3558186036 <== mon.1 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2163096241 0 0) 0x7f9ae41d7130 con 0x7f9ae41d7cd0 2024-03-29T11:20:51.617 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.623+0000 7f9ae17fa640 1 -- 172.21.15.176:0/3558186036 >> v1:172.21.15.114:6790/0 conn(0x7f9ae409ac40 legacy=0x7f9ae41d75c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:51.617 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.623+0000 7f9ae17fa640 1 -- 172.21.15.176:0/3558186036 >> v1:172.21.15.114:6789/0 conn(0x7f9ae40ad1e0 legacy=0x7f9ae4139400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:51.618 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.623+0000 7f9ae17fa640 10 monclient: _finish_hunting 0 2024-03-29T11:20:51.618 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.623+0000 7f9ae17fa640 1 monclient: found mon.b 2024-03-29T11:20:51.618 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.623+0000 7f9ae17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:51.618 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.623+0000 7f9ae17fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:20:51.618 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.623+0000 7f9ae17fa640 1 -- 172.21.15.176:0/3558186036 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ae40ae520 con 0x7f9ae41d7cd0 2024-03-29T11:20:51.618 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.623+0000 7f9ae17fa640 10 monclient: _finish_auth 0 2024-03-29T11:20:51.618 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.623+0000 7f9ae17fa640 10 monclient: _check_auth_tickets 2024-03-29T11:20:51.618 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.623+0000 7f9ae17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:51.618 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.623+0000 7f9aead54640 5 monclient: authenticate success, global_id 4153 2024-03-29T11:20:51.618 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.623+0000 7f9aead54640 10 monclient: _renew_subs 2024-03-29T11:20:51.618 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.623+0000 7f9aead54640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:20:51.618 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.623+0000 7f9aead54640 1 -- 172.21.15.176:0/3558186036 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ae40ae800 con 0x7f9ae41d7cd0 2024-03-29T11:20:51.618 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.623+0000 7f9aead54640 10 client.4153.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:51.618 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.623+0000 7f9aead54640 10 monclient: _renew_subs 2024-03-29T11:20:51.619 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.623+0000 7f9aead54640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:20:51.619 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.623+0000 7f9aead54640 1 -- 172.21.15.176:0/3558186036 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ae4142b70 con 0x7f9ae41d7cd0 2024-03-29T11:20:51.619 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.623+0000 7f9aead54640 1 librados: init done 2024-03-29T11:20:51.619 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.623+0000 7f9ae17fa640 1 -- 172.21.15.176:0/3558186036 <== mon.1 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9ad0001b80 con 0x7f9ae41d7cd0 2024-03-29T11:20:51.619 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.623+0000 7f9ae17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:51.619 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.623+0000 7f9ae17fa640 1 -- 172.21.15.176:0/3558186036 <== mon.1 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f9ad0003a60 con 0x7f9ae41d7cd0 2024-03-29T11:20:51.619 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.623+0000 7f9ae17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:51.619 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.623+0000 7f9ae17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:20:51.619 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.623+0000 7f9ae17fa640 10 monclient: dump: 2024-03-29T11:20:51.619 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-03-29T11:20:51.619 INFO:teuthology.orchestra.run.smithi176.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:20:51.619 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:51.619 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:51.620 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:51.620 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-03-29T11:20:51.620 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:20:51.620 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:20:51.620 INFO:teuthology.orchestra.run.smithi176.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:20:51.620 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T11:20:51.620 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.623+0000 7f9ae2ffd640 4 set_mon_vals no callback set 2024-03-29T11:20:51.620 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.623+0000 7f9ae17fa640 1 -- 172.21.15.176:0/3558186036 <== mon.1 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f9ad0004010 con 0x7f9ae41d7cd0 2024-03-29T11:20:51.620 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.623+0000 7f9ae17fa640 1 -- 172.21.15.176:0/3558186036 <== mon.1 v1:172.21.15.176:6789/0 8 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (unknown 1496670125 0 0) 0x7f9ad00069a0 con 0x7f9ae41d7cd0 2024-03-29T11:20:51.620 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.623+0000 7f9ae17fa640 10 client.4153.objecter ms_dispatch 0x7f9ae4000b90 osd_map(9..9 src has 1..9) v4 2024-03-29T11:20:51.620 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.623+0000 7f9ae17fa640 3 client.4153.objecter handle_osd_map got epochs [9,9] > 0 2024-03-29T11:20:51.620 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.623+0000 7f9ae17fa640 3 client.4153.objecter handle_osd_map decoding full epoch 9 2024-03-29T11:20:51.620 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.623+0000 7f9ae17fa640 20 client.4153.objecter dump_active .. 0 homeless 2024-03-29T11:20:51.620 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.624+0000 7f9aead54640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:51.620 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.624+0000 7f9aead54640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:51.621 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.624+0000 7f9aead54640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:20:51.621 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.624+0000 7f9aead54640 1 -- 172.21.15.176:0/3558186036 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ac4005080 con 0x7f9ae41d7cd0 2024-03-29T11:20:51.621 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.626+0000 7f9ae17fa640 1 -- 172.21.15.176:0/3558186036 <== mon.1 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f9ad0006340 con 0x7f9ae41d7cd0 2024-03-29T11:20:51.621 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.626+0000 7f9ae17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:51.621 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.626+0000 7f9ae17fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:51.638 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-03-29T11:20:51.662+0000 7fedaa420040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T11:20:51.674 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.698+0000 7f9aead54640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "fdb6a25f-6bd8-4c65-9b9b-b460ba7673a1", "id": 7}] 2024-03-29T11:20:51.675 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.698+0000 7f9aead54640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "fdb6a25f-6bd8-4c65-9b9b-b460ba7673a1", "id": 7}] 2024-03-29T11:20:51.675 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.698+0000 7f9aead54640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:20:51.675 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.698+0000 7f9aead54640 1 -- 172.21.15.176:0/3558186036 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd new", "uuid": "fdb6a25f-6bd8-4c65-9b9b-b460ba7673a1", "id": 7} v 0) v1 -- 0x7f9ac40058b0 con 0x7f9ae41d7cd0 2024-03-29T11:20:51.679 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.702+0000 7f9ae17fa640 1 -- 172.21.15.176:0/3558186036 <== mon.1 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "fdb6a25f-6bd8-4c65-9b9b-b460ba7673a1", "id": 7}]=0 v10) v1 ==== 112+0+2 (unknown 1195034965 0 1371665406) 0x7f9ad00065a0 con 0x7f9ae41d7cd0 2024-03-29T11:20:51.679 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.702+0000 7f9ae17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "fdb6a25f-6bd8-4c65-9b9b-b460ba7673a1", "id": 7}] 2024-03-29T11:20:51.679 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.702+0000 7f9ae17fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:51.679 INFO:teuthology.orchestra.run.smithi176.stdout:7 2024-03-29T11:20:51.680 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.703+0000 7f9aead54640 10 librados: watch_flush enter 2024-03-29T11:20:51.680 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.703+0000 7f9aead54640 10 librados: watch_flush exit 2024-03-29T11:20:51.680 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.703+0000 7f9aead54640 20 client.4153.objecter shutdown clearing up homeless session... 2024-03-29T11:20:51.680 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.703+0000 7f9aead54640 10 client.4153.objecter successfully canceled tick 2024-03-29T11:20:51.680 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.703+0000 7f9aead54640 10 monclient: shutdown 2024-03-29T11:20:51.680 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.703+0000 7f9aead54640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:51.681 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.703+0000 7f9aead54640 1 -- 172.21.15.176:0/3558186036 >> v1:172.21.15.176:6789/0 conn(0x7f9ae41d7cd0 legacy=0x7f9ae4138cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:51.681 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.703+0000 7f9aead54640 1 -- 172.21.15.176:0/3558186036 shutdown_connections 2024-03-29T11:20:51.681 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.703+0000 7f9aead54640 1 -- 172.21.15.176:0/3558186036 >> 172.21.15.176:0/3558186036 conn(0x7f9ae4066b00 msgr2=0x7f9ae40b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:51.681 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.703+0000 7f9aead54640 1 -- 172.21.15.176:0/3558186036 shutdown_connections 2024-03-29T11:20:51.681 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.703+0000 7f9aead54640 1 -- 172.21.15.176:0/3558186036 wait complete. 2024-03-29T11:20:51.681 INFO:teuthology.orchestra.run.smithi176.stderr:2024-03-29T11:20:51.703+0000 7f9aead54640 1 librados: shutdown 2024-03-29T11:20:51.681 INFO:tasks.ceph.mgr.y.smithi114.stderr:2024-03-29T11:20:51.703+0000 7ffbece52040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T11:20:51.690 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-29T11:20:51.690 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-03-29T11:20:51.693 INFO:tasks.ceph.osd.0:Started 2024-03-29T11:20:51.693 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-29T11:20:51.693 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-03-29T11:20:51.695 INFO:tasks.ceph.osd.1:Started 2024-03-29T11:20:51.695 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-29T11:20:51.695 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-03-29T11:20:51.697 INFO:tasks.ceph.osd.2:Started 2024-03-29T11:20:51.697 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-29T11:20:51.697 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-03-29T11:20:51.699 INFO:tasks.ceph.osd.3:Started 2024-03-29T11:20:51.699 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-29T11:20:51.699 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-03-29T11:20:51.701 INFO:tasks.ceph.osd.4:Started 2024-03-29T11:20:51.702 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-29T11:20:51.702 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-03-29T11:20:51.704 INFO:tasks.ceph.osd.5:Started 2024-03-29T11:20:51.704 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-29T11:20:51.704 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-03-29T11:20:51.706 INFO:tasks.ceph.osd.6:Started 2024-03-29T11:20:51.706 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-29T11:20:51.706 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-03-29T11:20:51.708 INFO:tasks.ceph.osd.7:Started 2024-03-29T11:20:51.709 DEBUG:teuthology.orchestra.run.smithi114:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T11:20:51.743 INFO:tasks.ceph.mgr.y.smithi114.stderr:2024-03-29T11:20:51.766+0000 7ffbece52040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T11:20:51.771 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-03-29T11:20:51.795+0000 7fedaa420040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T11:20:51.825 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-03-29T11:20:51.849+0000 7fedaa420040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T11:20:51.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.862+0000 7ff3e794d640 10 monclient: get_monmap_and_config 2024-03-29T11:20:51.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.862+0000 7ff3e794d640 10 monclient: build_initial_monmap 2024-03-29T11:20:51.839 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.862+0000 7ff3e794d640 10 monclient: monmap: 2024-03-29T11:20:51.839 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-03-29T11:20:51.839 INFO:teuthology.orchestra.run.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:20:51.839 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-03-29T11:20:51.863458+0000 2024-03-29T11:20:51.840 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-03-29T11:20:51.863458+0000 2024-03-29T11:20:51.840 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:51.840 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-03-29T11:20:51.840 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:20:51.840 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T11:20:51.840 INFO:teuthology.orchestra.run.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T11:20:51.840 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-03-29T11:20:51.840 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.862+0000 7ff3e794d640 1 Processor -- start 2024-03-29T11:20:51.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.863+0000 7ff3e794d640 1 -- start start 2024-03-29T11:20:51.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.863+0000 7ff3e794d640 10 monclient: init 2024-03-29T11:20:51.848 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.863+0000 7ff3e794d640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:51.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.863+0000 7ff3e794d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:20:51.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.863+0000 7ff3e794d640 10 monclient(hunting): picked mon.noname-c con 0x7ff3e00b8240 addr v1:172.21.15.176:6789/0 2024-03-29T11:20:51.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.863+0000 7ff3e794d640 10 monclient(hunting): picked mon.noname-b con 0x7ff3e00b1f60 addr v1:172.21.15.114:6790/0 2024-03-29T11:20:51.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.863+0000 7ff3e794d640 10 monclient(hunting): picked mon.noname-a con 0x7ff3e009c8c0 addr v1:172.21.15.114:6789/0 2024-03-29T11:20:51.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.863+0000 7ff3e794d640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3e009bd80 con 0x7ff3e009c8c0 2024-03-29T11:20:51.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.863+0000 7ff3e794d640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3e009b960 con 0x7ff3e00b8240 2024-03-29T11:20:51.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.863+0000 7ff3e794d640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3e00b2ec0 con 0x7ff3e00b1f60 2024-03-29T11:20:51.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.863+0000 7ff3e794d640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:51.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.863+0000 7ff3e794d640 10 monclient(hunting): authenticate will time out at 922.045918s 2024-03-29T11:20:51.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.863+0000 7ff3e4ee8640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7ff3e00b1f60 0x7ff3e009c1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:39356/0 (socket says 172.21.15.114:39356) 2024-03-29T11:20:51.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.863+0000 7ff3e4ee8640 1 -- 172.21.15.114:0/2862129009 learned_addr learned my addr 172.21.15.114:0/2862129009 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:20:51.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.864+0000 7ff3d7fff640 1 -- 172.21.15.114:0/2862129009 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1235475789 0 0) 0x7ff3e009bd80 con 0x7ff3e009c8c0 2024-03-29T11:20:51.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.864+0000 7ff3d7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:20:51.849 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.864+0000 7ff3d7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:51.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.864+0000 7ff3d7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:51.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.864+0000 7ff3d7fff640 10 monclient(hunting): my global_id is 4185 2024-03-29T11:20:51.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.864+0000 7ff3d7fff640 1 -- 172.21.15.114:0/2862129009 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3cc003520 con 0x7ff3e009c8c0 2024-03-29T11:20:51.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.864+0000 7ff3d7fff640 1 -- 172.21.15.114:0/2862129009 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1147136787 0 0) 0x7ff3e00b2ec0 con 0x7ff3e00b1f60 2024-03-29T11:20:51.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.864+0000 7ff3d7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:20:51.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.864+0000 7ff3d7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:51.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.864+0000 7ff3d7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:51.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.864+0000 7ff3d7fff640 10 monclient(hunting): my global_id is 4175 2024-03-29T11:20:51.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.864+0000 7ff3d7fff640 1 -- 172.21.15.114:0/2862129009 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3e009bd80 con 0x7ff3e00b1f60 2024-03-29T11:20:51.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.864+0000 7ff3d7fff640 1 -- 172.21.15.114:0/2862129009 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2800543004 0 0) 0x7ff3e009b960 con 0x7ff3e00b8240 2024-03-29T11:20:51.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.864+0000 7ff3d7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:20:51.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.864+0000 7ff3d7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:51.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.864+0000 7ff3d7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:51.850 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.864+0000 7ff3d7fff640 10 monclient(hunting): my global_id is 4177 2024-03-29T11:20:51.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.864+0000 7ff3d7fff640 1 -- 172.21.15.114:0/2862129009 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3e00b2ec0 con 0x7ff3e00b8240 2024-03-29T11:20:51.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.864+0000 7ff3d7fff640 1 -- 172.21.15.114:0/2862129009 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1862019766 0 0) 0x7ff3cc003520 con 0x7ff3e009c8c0 2024-03-29T11:20:51.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.864+0000 7ff3d7fff640 1 -- 172.21.15.114:0/2862129009 >> v1:172.21.15.114:6790/0 conn(0x7ff3e00b1f60 legacy=0x7ff3e009c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:51.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.864+0000 7ff3d7fff640 1 -- 172.21.15.114:0/2862129009 >> v1:172.21.15.176:6789/0 conn(0x7ff3e00b8240 legacy=0x7ff3e013f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:51.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.864+0000 7ff3d7fff640 10 monclient: _finish_hunting 0 2024-03-29T11:20:51.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.864+0000 7ff3d7fff640 1 monclient: found mon.noname-a 2024-03-29T11:20:51.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.864+0000 7ff3d7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:51.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.864+0000 7ff3d7fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.114:6789/0 2024-03-29T11:20:51.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.864+0000 7ff3d7fff640 1 -- 172.21.15.114:0/2862129009 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3e013fb60 con 0x7ff3e009c8c0 2024-03-29T11:20:51.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.864+0000 7ff3d7fff640 10 monclient: _finish_auth 0 2024-03-29T11:20:51.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.864+0000 7ff3d7fff640 10 monclient: _check_auth_tickets 2024-03-29T11:20:51.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.864+0000 7ff3d7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:51.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.864+0000 7ff3e794d640 5 monclient: authenticate success, global_id 4185 2024-03-29T11:20:51.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.864+0000 7ff3e794d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:51.851 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.864+0000 7ff3d7fff640 1 -- 172.21.15.114:0/2862129009 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7ff3d8001a10 con 0x7ff3e009c8c0 2024-03-29T11:20:51.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.864+0000 7ff3d7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:51.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.864+0000 7ff3d7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:20:51.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.864+0000 7ff3d7fff640 10 monclient: dump: 2024-03-29T11:20:51.852 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-03-29T11:20:51.852 INFO:teuthology.orchestra.run.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:20:51.852 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:51.852 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:51.852 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:51.852 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-03-29T11:20:51.852 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:20:51.852 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:20:51.852 INFO:teuthology.orchestra.run.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:20:51.852 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-03-29T11:20:51.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.864+0000 7ff3e794d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:51.852 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.864+0000 7ff3d7fff640 1 -- 172.21.15.114:0/2862129009 <== mon.0 v1:172.21.15.114:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff3d8001b70 con 0x7ff3e009c8c0 2024-03-29T11:20:51.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.864+0000 7ff3d7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:51.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.864+0000 7ff3e794d640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:51.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.864+0000 7ff3e794d640 4 set_mon_vals no callback set 2024-03-29T11:20:51.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.864+0000 7ff3d7fff640 1 -- 172.21.15.114:0/2862129009 <== mon.0 v1:172.21.15.114:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7ff3d8001fc0 con 0x7ff3e009c8c0 2024-03-29T11:20:51.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.864+0000 7ff3d7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:51.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.864+0000 7ff3d7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:20:51.853 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.864+0000 7ff3d7fff640 10 monclient: dump: 2024-03-29T11:20:51.853 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-03-29T11:20:51.853 INFO:teuthology.orchestra.run.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:20:51.853 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:51.853 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:51.853 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:51.853 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-03-29T11:20:51.854 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:20:51.854 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:20:51.854 INFO:teuthology.orchestra.run.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:20:51.854 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-03-29T11:20:51.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.865+0000 7ff3e794d640 10 monclient: shutdown 2024-03-29T11:20:51.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.865+0000 7ff3e794d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:51.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.865+0000 7ff3e794d640 1 -- 172.21.15.114:0/2862129009 >> v1:172.21.15.114:6789/0 conn(0x7ff3e009c8c0 legacy=0x7ff3e014d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:51.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.865+0000 7ff3e794d640 1 -- 172.21.15.114:0/2862129009 shutdown_connections 2024-03-29T11:20:51.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.865+0000 7ff3e794d640 1 -- 172.21.15.114:0/2862129009 >> 172.21.15.114:0/2862129009 conn(0x7ff3e0066b00 msgr2=0x7ff3e0066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:51.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.865+0000 7ff3e794d640 1 -- 172.21.15.114:0/2862129009 shutdown_connections 2024-03-29T11:20:51.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.865+0000 7ff3e794d640 1 -- 172.21.15.114:0/2862129009 wait complete. 2024-03-29T11:20:51.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.865+0000 7ff3e794d640 10 monclient: build_initial_monmap 2024-03-29T11:20:51.854 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.865+0000 7ff3e794d640 10 monclient: monmap: 2024-03-29T11:20:51.855 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 0 2024-03-29T11:20:51.855 INFO:teuthology.orchestra.run.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:51.855 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 0.000000 2024-03-29T11:20:51.855 INFO:teuthology.orchestra.run.smithi114.stderr:created 0.000000 2024-03-29T11:20:51.855 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:51.855 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-03-29T11:20:51.855 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:20:51.855 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T11:20:51.855 INFO:teuthology.orchestra.run.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T11:20:51.855 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-03-29T11:20:51.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.866+0000 7ff3e794d640 1 librados: starting msgr at 2024-03-29T11:20:51.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.866+0000 7ff3e794d640 1 librados: starting objecter 2024-03-29T11:20:51.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.866+0000 7ff3e794d640 1 Processor -- start 2024-03-29T11:20:51.855 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.866+0000 7ff3e794d640 1 -- start start 2024-03-29T11:20:51.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.866+0000 7ff3e794d640 1 librados: setting wanted keys 2024-03-29T11:20:51.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.866+0000 7ff3e794d640 1 librados: calling monclient init 2024-03-29T11:20:51.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.866+0000 7ff3e794d640 10 monclient: init 2024-03-29T11:20:51.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.866+0000 7ff3e794d640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:51.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.866+0000 7ff3e794d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:20:51.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.866+0000 7ff3e794d640 10 monclient(hunting): picked mon.noname-c con 0x7ff3e009c8c0 addr v1:172.21.15.114:6790/0 2024-03-29T11:20:51.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.866+0000 7ff3e794d640 10 monclient(hunting): picked mon.noname-a con 0x7ff3e00b8240 addr v1:172.21.15.114:6789/0 2024-03-29T11:20:51.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.866+0000 7ff3e794d640 10 monclient(hunting): picked mon.noname-b con 0x7ff3e0142250 addr v1:172.21.15.176:6789/0 2024-03-29T11:20:51.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.866+0000 7ff3e794d640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3e0142770 con 0x7ff3e00b8240 2024-03-29T11:20:51.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.867+0000 7ff3e4ee8640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7ff3e00b8240 0x7ff3e0140190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:52074/0 (socket says 172.21.15.114:52074) 2024-03-29T11:20:51.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.867+0000 7ff3e56e9640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7ff3e009c8c0 0x7ff3e00b6b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:39372/0 (socket says 172.21.15.114:39372) 2024-03-29T11:20:51.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.867+0000 7ff3e4ee8640 1 -- 172.21.15.114:0/924967693 learned_addr learned my addr 172.21.15.114:0/924967693 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:20:51.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.867+0000 7ff3d5ffb640 10 client.?.objecter ms_handle_connect 0x7ff3e00b8240 2024-03-29T11:20:51.856 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.867+0000 7ff3d5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:51.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.867+0000 7ff3d5ffb640 10 client.?.objecter ms_handle_connect 0x7ff3e009c8c0 2024-03-29T11:20:51.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.867+0000 7ff3d5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:51.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.867+0000 7ff3e794d640 1 -- 172.21.15.114:0/924967693 --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3e013c810 con 0x7ff3e0142250 2024-03-29T11:20:51.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.867+0000 7ff3e794d640 1 -- 172.21.15.114:0/924967693 --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3e013c9a0 con 0x7ff3e009c8c0 2024-03-29T11:20:51.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.867+0000 7ff3e794d640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:51.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.867+0000 7ff3e794d640 10 monclient(hunting): authenticate will time out at 922.049257s 2024-03-29T11:20:51.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.867+0000 7ff3d5ffb640 1 -- 172.21.15.114:0/924967693 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1437933750 0 0) 0x7ff3e0142770 con 0x7ff3e00b8240 2024-03-29T11:20:51.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.867+0000 7ff3d5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:20:51.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.867+0000 7ff3d5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:51.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.867+0000 7ff3d5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:51.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.867+0000 7ff3d5ffb640 10 monclient(hunting): my global_id is 4188 2024-03-29T11:20:51.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.867+0000 7ff3d5ffb640 1 -- 172.21.15.114:0/924967693 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3c4004eb0 con 0x7ff3e00b8240 2024-03-29T11:20:51.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.867+0000 7ff3d5ffb640 1 -- 172.21.15.114:0/924967693 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 393915307 0 0) 0x7ff3e013c9a0 con 0x7ff3e009c8c0 2024-03-29T11:20:51.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.867+0000 7ff3d5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:20:51.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.867+0000 7ff3d5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:51.857 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.867+0000 7ff3d5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:51.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.867+0000 7ff3d5ffb640 10 monclient(hunting): my global_id is 4178 2024-03-29T11:20:51.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.867+0000 7ff3d5ffb640 1 -- 172.21.15.114:0/924967693 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3e0142770 con 0x7ff3e009c8c0 2024-03-29T11:20:51.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.867+0000 7ff3d5ffb640 10 client.?.objecter ms_handle_connect 0x7ff3e0142250 2024-03-29T11:20:51.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.867+0000 7ff3d5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:51.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.867+0000 7ff3d5ffb640 1 -- 172.21.15.114:0/924967693 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2268447469 0 0) 0x7ff3c4004eb0 con 0x7ff3e00b8240 2024-03-29T11:20:51.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.867+0000 7ff3d5ffb640 1 -- 172.21.15.114:0/924967693 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff3c40054e0 con 0x7ff3e00b8240 2024-03-29T11:20:51.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.867+0000 7ff3d5ffb640 1 -- 172.21.15.114:0/924967693 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7ff3d0001a50 con 0x7ff3e00b8240 2024-03-29T11:20:51.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.867+0000 7ff3d5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:51.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.867+0000 7ff3d5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:20:51.858 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.867+0000 7ff3d5ffb640 10 monclient(hunting): dump: 2024-03-29T11:20:51.858 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-03-29T11:20:51.858 INFO:teuthology.orchestra.run.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:20:51.858 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:51.858 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:51.859 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:51.859 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-03-29T11:20:51.859 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:20:51.859 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:20:51.859 INFO:teuthology.orchestra.run.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:20:51.859 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-03-29T11:20:51.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.867+0000 7ff3d5ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:51.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.867+0000 7ff3d5ffb640 1 -- 172.21.15.114:0/924967693 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1724512870 0 0) 0x7ff3e013c810 con 0x7ff3e0142250 2024-03-29T11:20:51.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.867+0000 7ff3d5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:20:51.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.867+0000 7ff3d5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:51.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.867+0000 7ff3d5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:51.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.867+0000 7ff3d5ffb640 10 monclient(hunting): my global_id is 4180 2024-03-29T11:20:51.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.867+0000 7ff3d5ffb640 1 -- 172.21.15.114:0/924967693 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3c4004eb0 con 0x7ff3e0142250 2024-03-29T11:20:51.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.867+0000 7ff3d5ffb640 1 -- 172.21.15.114:0/924967693 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3597791376 0 0) 0x7ff3e0142770 con 0x7ff3e009c8c0 2024-03-29T11:20:51.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.867+0000 7ff3d5ffb640 1 -- 172.21.15.114:0/924967693 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff3e013c810 con 0x7ff3e009c8c0 2024-03-29T11:20:51.859 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.867+0000 7ff3d5ffb640 1 -- 172.21.15.114:0/924967693 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7ff3dc001870 con 0x7ff3e009c8c0 2024-03-29T11:20:51.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.867+0000 7ff3d5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:51.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.867+0000 7ff3d5ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:20:51.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.867+0000 7ff3d5ffb640 10 monclient(hunting): dump: 2024-03-29T11:20:51.860 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-03-29T11:20:51.860 INFO:teuthology.orchestra.run.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:20:51.860 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:51.860 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:51.860 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:51.860 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-03-29T11:20:51.860 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:20:51.860 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:20:51.860 INFO:teuthology.orchestra.run.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:20:51.860 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-03-29T11:20:51.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.867+0000 7ff3d5ffb640 1 -- 172.21.15.114:0/924967693 <== mon.0 v1:172.21.15.114:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2773565168 0 0) 0x7ff3c40054e0 con 0x7ff3e00b8240 2024-03-29T11:20:51.860 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.867+0000 7ff3d5ffb640 1 -- 172.21.15.114:0/924967693 >> v1:172.21.15.114:6790/0 conn(0x7ff3e009c8c0 legacy=0x7ff3e00b6b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:51.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.867+0000 7ff3d5ffb640 1 -- 172.21.15.114:0/924967693 >> v1:172.21.15.176:6789/0 conn(0x7ff3e0142250 legacy=0x7ff3e01408a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:51.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.867+0000 7ff3d5ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:20:51.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.867+0000 7ff3d5ffb640 1 monclient: found mon.a 2024-03-29T11:20:51.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.867+0000 7ff3d5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:51.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.867+0000 7ff3d5ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T11:20:51.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.867+0000 7ff3d5ffb640 1 -- 172.21.15.114:0/924967693 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3e00b7290 con 0x7ff3e00b8240 2024-03-29T11:20:51.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.867+0000 7ff3d5ffb640 10 monclient: _finish_auth 0 2024-03-29T11:20:51.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.867+0000 7ff3d5ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:20:51.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.867+0000 7ff3d5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:51.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.867+0000 7ff3e794d640 5 monclient: authenticate success, global_id 4188 2024-03-29T11:20:51.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.867+0000 7ff3e794d640 10 monclient: _renew_subs 2024-03-29T11:20:51.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.867+0000 7ff3e794d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T11:20:51.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.867+0000 7ff3e794d640 1 -- 172.21.15.114:0/924967693 --> v1:172.21.15.114:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3e00b7500 con 0x7ff3e00b8240 2024-03-29T11:20:51.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.867+0000 7ff3d5ffb640 1 -- 172.21.15.114:0/924967693 <== mon.0 v1:172.21.15.114:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff3d00047e0 con 0x7ff3e00b8240 2024-03-29T11:20:51.861 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.867+0000 7ff3d5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:51.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.867+0000 7ff3d5ffb640 1 -- 172.21.15.114:0/924967693 <== mon.0 v1:172.21.15.114:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7ff3d0004af0 con 0x7ff3e00b8240 2024-03-29T11:20:51.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.867+0000 7ff3d5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:51.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.867+0000 7ff3d5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:20:51.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.867+0000 7ff3d5ffb640 10 monclient: dump: 2024-03-29T11:20:51.862 INFO:teuthology.orchestra.run.smithi114.stderr:epoch 1 2024-03-29T11:20:51.862 INFO:teuthology.orchestra.run.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:20:51.862 INFO:teuthology.orchestra.run.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:51.862 INFO:teuthology.orchestra.run.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:51.862 INFO:teuthology.orchestra.run.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:51.862 INFO:teuthology.orchestra.run.smithi114.stderr:election_strategy: 1 2024-03-29T11:20:51.862 INFO:teuthology.orchestra.run.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:20:51.862 INFO:teuthology.orchestra.run.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:20:51.862 INFO:teuthology.orchestra.run.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:20:51.862 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-03-29T11:20:51.862 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.867+0000 7ff3d77fe640 4 set_mon_vals no callback set 2024-03-29T11:20:51.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.868+0000 7ff3e794d640 10 client.4188.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:51.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.868+0000 7ff3e794d640 10 monclient: _renew_subs 2024-03-29T11:20:51.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.868+0000 7ff3e794d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T11:20:51.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.868+0000 7ff3d5ffb640 1 -- 172.21.15.114:0/924967693 <== mon.0 v1:172.21.15.114:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7ff3d0005060 con 0x7ff3e00b8240 2024-03-29T11:20:51.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.868+0000 7ff3e794d640 1 -- 172.21.15.114:0/924967693 --> v1:172.21.15.114:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3e01e30f0 con 0x7ff3e00b8240 2024-03-29T11:20:51.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.868+0000 7ff3e794d640 1 librados: init done 2024-03-29T11:20:51.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.868+0000 7ff3d5ffb640 1 -- 172.21.15.114:0/924967693 <== mon.0 v1:172.21.15.114:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3238987313 0 0) 0x7ff3d00059e0 con 0x7ff3e00b8240 2024-03-29T11:20:51.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.868+0000 7ff3d5ffb640 10 client.4188.objecter ms_dispatch 0x7ff3e0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:20:51.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.868+0000 7ff3d5ffb640 3 client.4188.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:20:51.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.868+0000 7ff3d5ffb640 3 client.4188.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:20:51.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.868+0000 7ff3d5ffb640 20 client.4188.objecter dump_active .. 0 homeless 2024-03-29T11:20:51.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.868+0000 7ff3e794d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:51.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.868+0000 7ff3e794d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:51.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.868+0000 7ff3e794d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T11:20:51.863 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.868+0000 7ff3e794d640 1 -- 172.21.15.114:0/924967693 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3e009b150 con 0x7ff3e00b8240 2024-03-29T11:20:51.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.870+0000 7ff3d5ffb640 1 -- 172.21.15.114:0/924967693 <== mon.0 v1:172.21.15.114:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7ff3d0005ba0 con 0x7ff3e00b8240 2024-03-29T11:20:51.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.870+0000 7ff3d5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:51.864 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.870+0000 7ff3d5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:51.878 INFO:tasks.ceph.mgr.y.smithi114.stderr:2024-03-29T11:20:51.902+0000 7ffbece52040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:20:51.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.932+0000 7ff3e794d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:51.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.932+0000 7ff3e794d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:51.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.932+0000 7ff3e794d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T11:20:51.908 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.932+0000 7ff3e794d640 1 -- 172.21.15.114:0/924967693 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff3e004f730 con 0x7ff3e00b8240 2024-03-29T11:20:51.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.932+0000 7ff3d5ffb640 1 -- 172.21.15.114:0/924967693 <== mon.0 v1:172.21.15.114:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2858224613) 0x7ff3d0005420 con 0x7ff3e00b8240 2024-03-29T11:20:51.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.932+0000 7ff3d5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:51.909 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.932+0000 7ff3d5ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:51.909 INFO:teuthology.orchestra.run.smithi114.stdout: 2024-03-29T11:20:51.910 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":10,"fsid":"b0de2b9b-c45a-44c7-b6ee-e83b39a1e108","created":"2024-03-29T11:20:49.420660+0000","modified":"2024-03-29T11:20:51.700167+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:51.700167+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a64785d3-6b11-499a-b706-cc827ab96a74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5882dadb-e7a8-4604-ba9a-418013c106c8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d7e8c409-bc66-4e4a-bbdb-420bad31388e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0682989-3595-4cd6-90a5-ab9111c278e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"67068120-e3ff-4817-bd06-44feb7479a35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09e8b-8a85-4997-bdea-75d754e6ee67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2f0d9ead-f47f-4cb3-92a4-0a5ad23cd831","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fdb6a25f-6bd8-4c65-9b9b-b460ba7673a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:20:51.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.933+0000 7ff3e794d640 10 librados: watch_flush enter 2024-03-29T11:20:51.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.933+0000 7ff3e794d640 10 librados: watch_flush exit 2024-03-29T11:20:51.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.933+0000 7ff3e794d640 20 client.4188.objecter shutdown clearing up homeless session... 2024-03-29T11:20:51.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.933+0000 7ff3e794d640 10 client.4188.objecter successfully canceled tick 2024-03-29T11:20:51.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.933+0000 7ff3e794d640 10 monclient: shutdown 2024-03-29T11:20:51.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.933+0000 7ff3e794d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:51.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.933+0000 7ff3e794d640 1 -- 172.21.15.114:0/924967693 >> v1:172.21.15.114:6789/0 conn(0x7ff3e00b8240 legacy=0x7ff3e0140190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:51.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.933+0000 7ff3e794d640 1 -- 172.21.15.114:0/924967693 shutdown_connections 2024-03-29T11:20:51.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.933+0000 7ff3e794d640 1 -- 172.21.15.114:0/924967693 >> 172.21.15.114:0/924967693 conn(0x7ff3e0066b00 msgr2=0x7ff3e009c1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:51.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.933+0000 7ff3e794d640 1 -- 172.21.15.114:0/924967693 shutdown_connections 2024-03-29T11:20:51.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.933+0000 7ff3e794d640 1 -- 172.21.15.114:0/924967693 wait complete. 2024-03-29T11:20:51.911 INFO:teuthology.orchestra.run.smithi114.stderr:2024-03-29T11:20:51.933+0000 7ff3e794d640 1 librados: shutdown 2024-03-29T11:20:51.920 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-29T11:20:51.920 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-29T11:20:51.938 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-03-29T11:20:51.961+0000 7fedaa420040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:20:51.977 INFO:tasks.ceph.mgr.y.smithi114.stderr:2024-03-29T11:20:52.000+0000 7ffbece52040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T11:20:52.034 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-03-29T11:20:52.057+0000 7fedaa420040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T11:20:52.134 INFO:tasks.ceph.mgr.y.smithi114.stderr:2024-03-29T11:20:52.157+0000 7ffbece52040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:20:52.176 INFO:tasks.ceph.mgr.y.smithi114.stderr:2024-03-29T11:20:52.199+0000 7ffbece52040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T11:20:52.190 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-03-29T11:20:52.213+0000 7fedaa420040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:20:52.233 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-03-29T11:20:52.256+0000 7fedaa420040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T11:20:52.292 INFO:tasks.ceph.osd.2.smithi114.stderr:2024-03-29T11:20:52.315+0000 7f94d1215640 -1 Falling back to public interface 2024-03-29T11:20:52.293 INFO:tasks.ceph.osd.1.smithi114.stderr:2024-03-29T11:20:52.316+0000 7efdcc211640 -1 Falling back to public interface 2024-03-29T11:20:52.312 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-03-29T11:20:52.335+0000 7f0f41011640 -1 Falling back to public interface 2024-03-29T11:20:52.321 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-03-29T11:20:52.344+0000 7f222a311640 -1 Falling back to public interface 2024-03-29T11:20:52.326 INFO:tasks.ceph.osd.0.smithi114.stderr:2024-03-29T11:20:52.349+0000 7f5ff939e640 -1 Falling back to public interface 2024-03-29T11:20:52.326 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-03-29T11:20:52.350+0000 7ff79560c640 -1 Falling back to public interface 2024-03-29T11:20:52.329 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-03-29T11:20:52.352+0000 7f487a012640 -1 Falling back to public interface 2024-03-29T11:20:52.332 INFO:tasks.ceph.mgr.y.smithi114.stderr:2024-03-29T11:20:52.356+0000 7ffbece52040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T11:20:52.332 INFO:tasks.ceph.osd.3.smithi114.stderr:2024-03-29T11:20:52.356+0000 7fe5efa0d640 -1 Falling back to public interface 2024-03-29T11:20:52.391 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-03-29T11:20:52.414+0000 7fedaa420040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T11:20:52.726 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:20:52.818 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.841+0000 7ff8d47d6640 10 monclient: get_monmap_and_config 2024-03-29T11:20:52.818 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.841+0000 7ff8d47d6640 10 monclient: build_initial_monmap 2024-03-29T11:20:52.818 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.841+0000 7ff8d47d6640 10 monclient: monmap: 2024-03-29T11:20:52.818 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:20:52.818 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:20:52.818 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:52.842467+0000 2024-03-29T11:20:52.818 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:52.842467+0000 2024-03-29T11:20:52.818 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:52.818 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:20:52.818 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:20:52.818 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T11:20:52.819 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T11:20:52.819 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:20:52.826 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.841+0000 7ff8d47d6640 1 Processor -- start 2024-03-29T11:20:52.827 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.842+0000 7ff8d47d6640 1 -- start start 2024-03-29T11:20:52.827 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.842+0000 7ff8d47d6640 10 monclient: init 2024-03-29T11:20:52.827 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.842+0000 7ff8d47d6640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:52.827 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.842+0000 7ff8d47d6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:20:52.827 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.842+0000 7ff8d47d6640 10 monclient(hunting): picked mon.noname-a con 0x7ff8cc1422b0 addr v1:172.21.15.114:6789/0 2024-03-29T11:20:52.827 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.842+0000 7ff8d47d6640 10 monclient(hunting): picked mon.noname-c con 0x7ff8cc142690 addr v1:172.21.15.176:6789/0 2024-03-29T11:20:52.827 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.842+0000 7ff8d47d6640 10 monclient(hunting): picked mon.noname-b con 0x7ff8cc0b17d0 addr v1:172.21.15.114:6790/0 2024-03-29T11:20:52.827 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.842+0000 7ff8d47d6640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8cc142c80 con 0x7ff8cc1422b0 2024-03-29T11:20:52.827 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.842+0000 7ff8d47d6640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8cc0aae40 con 0x7ff8cc142690 2024-03-29T11:20:52.827 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.842+0000 7ff8d47d6640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8cc13fbd0 con 0x7ff8cc0b17d0 2024-03-29T11:20:52.827 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.842+0000 7ff8d47d6640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:52.827 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.842+0000 7ff8d47d6640 10 monclient(hunting): authenticate will time out at 923.024909s 2024-03-29T11:20:52.827 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.842+0000 7ff8d2572640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7ff8cc1422b0 0x7ff8cc13f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:48348/0 (socket says 172.21.15.114:48348) 2024-03-29T11:20:52.827 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.842+0000 7ff8d2d73640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7ff8cc0b17d0 0x7ff8cc0b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:41262/0 (socket says 172.21.15.114:41262) 2024-03-29T11:20:52.828 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.842+0000 7ff8d2572640 1 -- 172.21.15.114:0/2915153570 learned_addr learned my addr 172.21.15.114:0/2915153570 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:20:52.828 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.843+0000 7ff8d1570640 1 -- 172.21.15.114:0/2915153570 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3232310281 0 0) 0x7ff8cc13fbd0 con 0x7ff8cc0b17d0 2024-03-29T11:20:52.828 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.843+0000 7ff8d1570640 10 monclient(hunting): _negotiate 2024-03-29T11:20:52.828 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.843+0000 7ff8d1570640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:52.828 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.843+0000 7ff8d1570640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:52.828 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.843+0000 7ff8d1570640 10 monclient(hunting): my global_id is 4181 2024-03-29T11:20:52.828 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.843+0000 7ff8d1570640 1 -- 172.21.15.114:0/2915153570 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8b8003520 con 0x7ff8cc0b17d0 2024-03-29T11:20:52.828 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.843+0000 7ff8d1570640 1 -- 172.21.15.114:0/2915153570 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 802018323 0 0) 0x7ff8cc142c80 con 0x7ff8cc1422b0 2024-03-29T11:20:52.828 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.843+0000 7ff8d1570640 10 monclient(hunting): _negotiate 2024-03-29T11:20:52.828 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.843+0000 7ff8d1570640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:52.828 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.843+0000 7ff8d1570640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:52.828 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.843+0000 7ff8d1570640 10 monclient(hunting): my global_id is 4191 2024-03-29T11:20:52.828 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.843+0000 7ff8d1570640 1 -- 172.21.15.114:0/2915153570 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8cc13fbd0 con 0x7ff8cc1422b0 2024-03-29T11:20:52.828 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.843+0000 7ff8d1570640 1 -- 172.21.15.114:0/2915153570 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1388752182 0 0) 0x7ff8cc0aae40 con 0x7ff8cc142690 2024-03-29T11:20:52.828 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.843+0000 7ff8d1570640 10 monclient(hunting): _negotiate 2024-03-29T11:20:52.829 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.843+0000 7ff8d1570640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:52.829 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.843+0000 7ff8d1570640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:52.829 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.843+0000 7ff8d1570640 10 monclient(hunting): my global_id is 4183 2024-03-29T11:20:52.829 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.843+0000 7ff8d1570640 1 -- 172.21.15.114:0/2915153570 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8cc142c80 con 0x7ff8cc142690 2024-03-29T11:20:52.829 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.843+0000 7ff8d1570640 1 -- 172.21.15.114:0/2915153570 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 708174102 0 0) 0x7ff8b8003520 con 0x7ff8cc0b17d0 2024-03-29T11:20:52.829 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.843+0000 7ff8d1570640 1 -- 172.21.15.114:0/2915153570 >> v1:172.21.15.176:6789/0 conn(0x7ff8cc142690 legacy=0x7ff8cc0b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:52.829 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.843+0000 7ff8d1570640 1 -- 172.21.15.114:0/2915153570 >> v1:172.21.15.114:6789/0 conn(0x7ff8cc1422b0 legacy=0x7ff8cc13f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:52.829 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.843+0000 7ff8d1570640 10 monclient: _finish_hunting 0 2024-03-29T11:20:52.829 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.843+0000 7ff8d1570640 1 monclient: found mon.noname-b 2024-03-29T11:20:52.829 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.843+0000 7ff8d1570640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:52.829 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.843+0000 7ff8d1570640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.114:6790/0 2024-03-29T11:20:52.829 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.843+0000 7ff8d1570640 1 -- 172.21.15.114:0/2915153570 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8cc151940 con 0x7ff8cc0b17d0 2024-03-29T11:20:52.830 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.843+0000 7ff8d1570640 10 monclient: _finish_auth 0 2024-03-29T11:20:52.830 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.843+0000 7ff8d1570640 10 monclient: _check_auth_tickets 2024-03-29T11:20:52.830 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.843+0000 7ff8d1570640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:52.830 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.843+0000 7ff8d1570640 1 -- 172.21.15.114:0/2915153570 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7ff8c4001940 con 0x7ff8cc0b17d0 2024-03-29T11:20:52.830 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.843+0000 7ff8d47d6640 5 monclient: authenticate success, global_id 4181 2024-03-29T11:20:52.830 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.843+0000 7ff8d47d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:52.830 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.843+0000 7ff8d1570640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:52.830 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.843+0000 7ff8d1570640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:20:52.830 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.843+0000 7ff8d1570640 10 monclient: dump: 2024-03-29T11:20:52.830 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:20:52.830 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:20:52.830 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:52.830 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:52.830 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:52.830 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:20:52.831 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:20:52.831 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:20:52.831 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:20:52.831 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:20:52.831 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.843+0000 7ff8d47d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:52.831 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.843+0000 7ff8d1570640 1 -- 172.21.15.114:0/2915153570 <== mon.2 v1:172.21.15.114:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff8c4001aa0 con 0x7ff8cc0b17d0 2024-03-29T11:20:52.831 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.843+0000 7ff8d1570640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:52.831 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.843+0000 7ff8d1570640 1 -- 172.21.15.114:0/2915153570 <== mon.2 v1:172.21.15.114:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7ff8c4001e70 con 0x7ff8cc0b17d0 2024-03-29T11:20:52.831 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.843+0000 7ff8d47d6640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:52.831 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.843+0000 7ff8d47d6640 4 set_mon_vals no callback set 2024-03-29T11:20:52.831 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.843+0000 7ff8d1570640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:52.831 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.843+0000 7ff8d1570640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:20:52.831 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.843+0000 7ff8d1570640 10 monclient: dump: 2024-03-29T11:20:52.831 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:20:52.831 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:20:52.832 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:52.832 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:52.832 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:52.832 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:20:52.832 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:20:52.832 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:20:52.832 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:20:52.832 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:20:52.832 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.844+0000 7ff8d47d6640 10 monclient: shutdown 2024-03-29T11:20:52.832 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.844+0000 7ff8d47d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:52.832 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.844+0000 7ff8d47d6640 1 -- 172.21.15.114:0/2915153570 >> v1:172.21.15.114:6790/0 conn(0x7ff8cc0b17d0 legacy=0x7ff8cc0b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:52.832 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.844+0000 7ff8d47d6640 1 -- 172.21.15.114:0/2915153570 shutdown_connections 2024-03-29T11:20:52.832 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.844+0000 7ff8d47d6640 1 -- 172.21.15.114:0/2915153570 >> 172.21.15.114:0/2915153570 conn(0x7ff8cc066a00 msgr2=0x7ff8cc066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:52.832 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.844+0000 7ff8d47d6640 1 -- 172.21.15.114:0/2915153570 shutdown_connections 2024-03-29T11:20:52.833 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.844+0000 7ff8d47d6640 1 -- 172.21.15.114:0/2915153570 wait complete. 2024-03-29T11:20:52.833 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.844+0000 7ff8d47d6640 10 monclient: build_initial_monmap 2024-03-29T11:20:52.833 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.844+0000 7ff8d47d6640 10 monclient: monmap: 2024-03-29T11:20:52.833 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:20:52.833 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:52.833 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T11:20:52.833 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T11:20:52.833 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:52.833 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:20:52.833 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:20:52.833 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T11:20:52.833 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T11:20:52.833 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:20:52.833 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.844+0000 7ff8d47d6640 1 librados: starting msgr at 2024-03-29T11:20:52.833 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.844+0000 7ff8d47d6640 1 librados: starting objecter 2024-03-29T11:20:52.833 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.845+0000 7ff8d47d6640 1 Processor -- start 2024-03-29T11:20:52.834 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.845+0000 7ff8d47d6640 1 -- start start 2024-03-29T11:20:52.834 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.845+0000 7ff8d47d6640 1 librados: setting wanted keys 2024-03-29T11:20:52.834 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.845+0000 7ff8d47d6640 1 librados: calling monclient init 2024-03-29T11:20:52.834 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.845+0000 7ff8d47d6640 10 monclient: init 2024-03-29T11:20:52.834 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.845+0000 7ff8d47d6640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:52.834 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.845+0000 7ff8d47d6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:20:52.834 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.845+0000 7ff8d47d6640 10 monclient(hunting): picked mon.noname-c con 0x7ff8cc0b17d0 addr v1:172.21.15.114:6790/0 2024-03-29T11:20:52.834 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.845+0000 7ff8d47d6640 10 monclient(hunting): picked mon.noname-a con 0x7ff8cc1422b0 addr v1:172.21.15.114:6789/0 2024-03-29T11:20:52.834 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.845+0000 7ff8d47d6640 10 monclient(hunting): picked mon.noname-b con 0x7ff8cc142690 addr v1:172.21.15.176:6789/0 2024-03-29T11:20:52.834 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.845+0000 7ff8d47d6640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8cc0abaf0 con 0x7ff8cc1422b0 2024-03-29T11:20:52.834 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.845+0000 7ff8d47d6640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8cc1323c0 con 0x7ff8cc142690 2024-03-29T11:20:52.834 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.845+0000 7ff8d47d6640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8cc0abf80 con 0x7ff8cc0b17d0 2024-03-29T11:20:52.834 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.845+0000 7ff8d47d6640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:52.834 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.845+0000 7ff8d47d6640 10 monclient(hunting): authenticate will time out at 923.027776s 2024-03-29T11:20:52.835 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.845+0000 7ff8d1d71640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7ff8cc1422b0 0x7ff8cc12e490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:48358/0 (socket says 172.21.15.114:48358) 2024-03-29T11:20:52.835 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.845+0000 7ff8d2572640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7ff8cc0b17d0 0x7ff8cc129f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:41278/0 (socket says 172.21.15.114:41278) 2024-03-29T11:20:52.835 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.845+0000 7ff8d1d71640 1 -- 172.21.15.114:0/2520655141 learned_addr learned my addr 172.21.15.114:0/2520655141 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:20:52.835 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.845+0000 7ff8c2ffd640 10 client.?.objecter ms_handle_connect 0x7ff8cc1422b0 2024-03-29T11:20:52.835 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.845+0000 7ff8c2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:52.835 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.845+0000 7ff8c2ffd640 10 client.?.objecter ms_handle_connect 0x7ff8cc0b17d0 2024-03-29T11:20:52.835 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.845+0000 7ff8c2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:52.835 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.845+0000 7ff8c2ffd640 1 -- 172.21.15.114:0/2520655141 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 157326979 0 0) 0x7ff8cc0abaf0 con 0x7ff8cc1422b0 2024-03-29T11:20:52.835 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.845+0000 7ff8c2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:20:52.835 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.845+0000 7ff8c2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:52.835 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.845+0000 7ff8c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:52.835 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.845+0000 7ff8c2ffd640 10 monclient(hunting): my global_id is 4194 2024-03-29T11:20:52.835 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.845+0000 7ff8c2ffd640 1 -- 172.21.15.114:0/2520655141 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8b0004eb0 con 0x7ff8cc1422b0 2024-03-29T11:20:52.835 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.845+0000 7ff8c2ffd640 10 client.?.objecter ms_handle_connect 0x7ff8cc142690 2024-03-29T11:20:52.835 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.845+0000 7ff8c2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:52.836 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.845+0000 7ff8c2ffd640 1 -- 172.21.15.114:0/2520655141 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3723470806 0 0) 0x7ff8cc0abf80 con 0x7ff8cc0b17d0 2024-03-29T11:20:52.836 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.845+0000 7ff8c2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:20:52.836 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.845+0000 7ff8c2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:52.836 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.845+0000 7ff8c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:52.836 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.845+0000 7ff8c2ffd640 10 monclient(hunting): my global_id is 4184 2024-03-29T11:20:52.836 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.845+0000 7ff8c2ffd640 1 -- 172.21.15.114:0/2520655141 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8cc0abaf0 con 0x7ff8cc0b17d0 2024-03-29T11:20:52.836 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.846+0000 7ff8c2ffd640 1 -- 172.21.15.114:0/2520655141 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4131405052 0 0) 0x7ff8cc1323c0 con 0x7ff8cc142690 2024-03-29T11:20:52.836 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.846+0000 7ff8c2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:20:52.836 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.846+0000 7ff8c2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:52.836 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.846+0000 7ff8c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:52.836 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.846+0000 7ff8c2ffd640 10 monclient(hunting): my global_id is 4186 2024-03-29T11:20:52.836 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.846+0000 7ff8c2ffd640 1 -- 172.21.15.114:0/2520655141 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8cc0abf80 con 0x7ff8cc142690 2024-03-29T11:20:52.836 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.846+0000 7ff8c2ffd640 1 -- 172.21.15.114:0/2520655141 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 542119553 0 0) 0x7ff8b0004eb0 con 0x7ff8cc1422b0 2024-03-29T11:20:52.836 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.846+0000 7ff8c2ffd640 1 -- 172.21.15.114:0/2520655141 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff8cc1323c0 con 0x7ff8cc1422b0 2024-03-29T11:20:52.836 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.846+0000 7ff8c2ffd640 1 -- 172.21.15.114:0/2520655141 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7ff8c8001970 con 0x7ff8cc1422b0 2024-03-29T11:20:52.837 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.846+0000 7ff8c2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:52.837 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.846+0000 7ff8c2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:20:52.837 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.846+0000 7ff8c2ffd640 10 monclient(hunting): dump: 2024-03-29T11:20:52.837 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:20:52.837 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:20:52.837 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:52.837 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:52.837 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:52.837 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:20:52.837 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:20:52.837 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:20:52.837 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:20:52.837 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:20:52.837 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.846+0000 7ff8c2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:52.837 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.846+0000 7ff8c2ffd640 1 -- 172.21.15.114:0/2520655141 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 860903876 0 0) 0x7ff8cc0abaf0 con 0x7ff8cc0b17d0 2024-03-29T11:20:52.837 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.846+0000 7ff8c2ffd640 1 -- 172.21.15.114:0/2520655141 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff8b0004eb0 con 0x7ff8cc0b17d0 2024-03-29T11:20:52.838 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.846+0000 7ff8c2ffd640 1 -- 172.21.15.114:0/2520655141 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7ff8bc001250 con 0x7ff8cc0b17d0 2024-03-29T11:20:52.838 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.846+0000 7ff8c2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:52.838 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.846+0000 7ff8c2ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:20:52.838 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.846+0000 7ff8c2ffd640 10 monclient(hunting): dump: 2024-03-29T11:20:52.838 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:20:52.838 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:20:52.838 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:52.838 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:52.838 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:52.838 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:20:52.838 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:20:52.838 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:20:52.838 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:20:52.838 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:20:52.839 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.846+0000 7ff8c2ffd640 1 -- 172.21.15.114:0/2520655141 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1292644892 0 0) 0x7ff8cc0abf80 con 0x7ff8cc142690 2024-03-29T11:20:52.839 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.846+0000 7ff8c2ffd640 1 -- 172.21.15.114:0/2520655141 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff8cc0abaf0 con 0x7ff8cc142690 2024-03-29T11:20:52.839 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.846+0000 7ff8c2ffd640 1 -- 172.21.15.114:0/2520655141 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7ff8c4001760 con 0x7ff8cc142690 2024-03-29T11:20:52.839 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.846+0000 7ff8c2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:52.839 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.846+0000 7ff8c2ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:20:52.839 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.846+0000 7ff8c2ffd640 10 monclient(hunting): dump: 2024-03-29T11:20:52.839 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:20:52.839 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:20:52.839 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:52.839 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:52.839 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:52.839 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:20:52.839 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:20:52.839 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:20:52.839 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:20:52.840 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:20:52.840 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.846+0000 7ff8c2ffd640 1 -- 172.21.15.114:0/2520655141 <== mon.0 v1:172.21.15.114:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3725507930 0 0) 0x7ff8cc1323c0 con 0x7ff8cc1422b0 2024-03-29T11:20:52.840 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.846+0000 7ff8c2ffd640 1 -- 172.21.15.114:0/2520655141 >> v1:172.21.15.114:6790/0 conn(0x7ff8cc0b17d0 legacy=0x7ff8cc129f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:52.840 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.846+0000 7ff8c2ffd640 1 -- 172.21.15.114:0/2520655141 >> v1:172.21.15.176:6789/0 conn(0x7ff8cc142690 legacy=0x7ff8cc1e2d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:52.840 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.846+0000 7ff8c2ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:20:52.840 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.846+0000 7ff8c2ffd640 1 monclient: found mon.a 2024-03-29T11:20:52.840 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.846+0000 7ff8c2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:52.840 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.846+0000 7ff8c2ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T11:20:52.840 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.846+0000 7ff8c2ffd640 1 -- 172.21.15.114:0/2520655141 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8cc131f30 con 0x7ff8cc1422b0 2024-03-29T11:20:52.840 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.846+0000 7ff8c2ffd640 10 monclient: _finish_auth 0 2024-03-29T11:20:52.840 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.846+0000 7ff8c2ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:20:52.840 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.846+0000 7ff8c2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:52.840 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.846+0000 7ff8d47d6640 5 monclient: authenticate success, global_id 4194 2024-03-29T11:20:52.840 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.846+0000 7ff8d47d6640 10 monclient: _renew_subs 2024-03-29T11:20:52.840 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.846+0000 7ff8d47d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T11:20:52.841 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.846+0000 7ff8d47d6640 1 -- 172.21.15.114:0/2520655141 --> v1:172.21.15.114:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8cc132180 con 0x7ff8cc1422b0 2024-03-29T11:20:52.841 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.846+0000 7ff8d47d6640 10 client.4194.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:52.841 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.846+0000 7ff8d47d6640 10 monclient: _renew_subs 2024-03-29T11:20:52.841 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.846+0000 7ff8d47d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T11:20:52.841 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.846+0000 7ff8d47d6640 1 -- 172.21.15.114:0/2520655141 --> v1:172.21.15.114:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8cc0ab720 con 0x7ff8cc1422b0 2024-03-29T11:20:52.841 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.846+0000 7ff8d47d6640 1 librados: init done 2024-03-29T11:20:52.841 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.846+0000 7ff8c2ffd640 1 -- 172.21.15.114:0/2520655141 <== mon.0 v1:172.21.15.114:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff8c8001c50 con 0x7ff8cc1422b0 2024-03-29T11:20:52.841 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.846+0000 7ff8c2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:52.841 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.846+0000 7ff8c2ffd640 1 -- 172.21.15.114:0/2520655141 <== mon.0 v1:172.21.15.114:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7ff8c8003a60 con 0x7ff8cc1422b0 2024-03-29T11:20:52.841 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.846+0000 7ff8c2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:52.841 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.846+0000 7ff8c2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:20:52.841 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.846+0000 7ff8c2ffd640 10 monclient: dump: 2024-03-29T11:20:52.842 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:20:52.842 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:20:52.842 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:52.842 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:52.842 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:52.842 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:20:52.842 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:20:52.842 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:20:52.842 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:20:52.842 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:20:52.842 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.846+0000 7ff8d0d6f640 4 set_mon_vals no callback set 2024-03-29T11:20:52.842 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.846+0000 7ff8c2ffd640 1 -- 172.21.15.114:0/2520655141 <== mon.0 v1:172.21.15.114:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7ff8c8003fd0 con 0x7ff8cc1422b0 2024-03-29T11:20:52.843 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.846+0000 7ff8c2ffd640 1 -- 172.21.15.114:0/2520655141 <== mon.0 v1:172.21.15.114:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3238987313 0 0) 0x7ff8c8004280 con 0x7ff8cc1422b0 2024-03-29T11:20:52.843 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.846+0000 7ff8c2ffd640 10 client.4194.objecter ms_dispatch 0x7ff8cc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:20:52.843 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.846+0000 7ff8c2ffd640 3 client.4194.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:20:52.843 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.846+0000 7ff8c2ffd640 3 client.4194.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:20:52.843 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.846+0000 7ff8c2ffd640 20 client.4194.objecter dump_active .. 0 homeless 2024-03-29T11:20:52.843 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.847+0000 7ff8d47d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:52.843 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.847+0000 7ff8d47d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:52.843 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.847+0000 7ff8d47d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T11:20:52.843 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.847+0000 7ff8d47d6640 1 -- 172.21.15.114:0/2520655141 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8cc0abc80 con 0x7ff8cc1422b0 2024-03-29T11:20:52.843 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.849+0000 7ff8c2ffd640 1 -- 172.21.15.114:0/2520655141 <== mon.0 v1:172.21.15.114:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7ff8c80057c0 con 0x7ff8cc1422b0 2024-03-29T11:20:52.843 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.849+0000 7ff8c2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:52.843 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.849+0000 7ff8c2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:52.890 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.914+0000 7ff8d47d6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:52.891 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.914+0000 7ff8d47d6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:52.891 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.914+0000 7ff8d47d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T11:20:52.891 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.914+0000 7ff8d47d6640 1 -- 172.21.15.114:0/2520655141 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff8cc075f10 con 0x7ff8cc1422b0 2024-03-29T11:20:52.891 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.914+0000 7ff8c2ffd640 1 -- 172.21.15.114:0/2520655141 <== mon.0 v1:172.21.15.114:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2858224613) 0x7ff8c80059e0 con 0x7ff8cc1422b0 2024-03-29T11:20:52.892 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.914+0000 7ff8c2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:52.892 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.914+0000 7ff8c2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:52.892 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T11:20:52.892 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"b0de2b9b-c45a-44c7-b6ee-e83b39a1e108","created":"2024-03-29T11:20:49.420660+0000","modified":"2024-03-29T11:20:51.700167+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:51.700167+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a64785d3-6b11-499a-b706-cc827ab96a74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5882dadb-e7a8-4604-ba9a-418013c106c8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d7e8c409-bc66-4e4a-bbdb-420bad31388e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0682989-3595-4cd6-90a5-ab9111c278e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"67068120-e3ff-4817-bd06-44feb7479a35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09e8b-8a85-4997-bdea-75d754e6ee67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2f0d9ead-f47f-4cb3-92a4-0a5ad23cd831","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fdb6a25f-6bd8-4c65-9b9b-b460ba7673a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:20:52.893 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.915+0000 7ff8d47d6640 10 librados: watch_flush enter 2024-03-29T11:20:52.893 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.915+0000 7ff8d47d6640 10 librados: watch_flush exit 2024-03-29T11:20:52.893 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.915+0000 7ff8d47d6640 20 client.4194.objecter shutdown clearing up homeless session... 2024-03-29T11:20:52.893 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.915+0000 7ff8d47d6640 10 client.4194.objecter successfully canceled tick 2024-03-29T11:20:52.894 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.915+0000 7ff8d47d6640 10 monclient: shutdown 2024-03-29T11:20:52.894 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.915+0000 7ff8d47d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:52.894 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.915+0000 7ff8d47d6640 1 -- 172.21.15.114:0/2520655141 >> v1:172.21.15.114:6789/0 conn(0x7ff8cc1422b0 legacy=0x7ff8cc12e490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:52.894 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.915+0000 7ff8d47d6640 1 -- 172.21.15.114:0/2520655141 shutdown_connections 2024-03-29T11:20:52.894 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.915+0000 7ff8d47d6640 1 -- 172.21.15.114:0/2520655141 >> 172.21.15.114:0/2520655141 conn(0x7ff8cc066a00 msgr2=0x7ff8cc0b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:52.894 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.916+0000 7ff8d47d6640 1 -- 172.21.15.114:0/2520655141 shutdown_connections 2024-03-29T11:20:52.894 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.916+0000 7ff8d47d6640 1 -- 172.21.15.114:0/2520655141 wait complete. 2024-03-29T11:20:52.894 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:52.916+0000 7ff8d47d6640 1 librados: shutdown 2024-03-29T11:20:52.903 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:20:53.161 INFO:tasks.ceph.mgr.y.smithi114.stderr:2024-03-29T11:20:53.185+0000 7ffbece52040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T11:20:53.213 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-03-29T11:20:53.236+0000 7fedaa420040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T11:20:53.441 INFO:tasks.ceph.mgr.y.smithi114.stderr:2024-03-29T11:20:53.464+0000 7ffbece52040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T11:20:53.491 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-03-29T11:20:53.514+0000 7fedaa420040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T11:20:53.492 INFO:tasks.ceph.mgr.y.smithi114.stderr:2024-03-29T11:20:53.515+0000 7ffbece52040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T11:20:53.541 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-03-29T11:20:53.564+0000 7fedaa420040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T11:20:53.607 INFO:tasks.ceph.mgr.y.smithi114.stderr:2024-03-29T11:20:53.630+0000 7ffbece52040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T11:20:53.659 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-03-29T11:20:53.682+0000 7fedaa420040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T11:20:54.620 INFO:tasks.ceph.mgr.y.smithi114.stderr:2024-03-29T11:20:54.644+0000 7ffb8db91640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-29T11:20:54.621 INFO:tasks.ceph.mgr.y.smithi114.stderr:2024-03-29T11:20:54.644+0000 7ffb8db91640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-29T11:20:55.775 INFO:tasks.ceph.osd.2.smithi114.stderr:2024-03-29T11:20:55.799+0000 7f94d1215640 -1 osd.2 0 log_to_monitors true 2024-03-29T11:20:55.800 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-03-29T11:20:55.823+0000 7f0f41011640 -1 osd.5 0 log_to_monitors true 2024-03-29T11:20:55.807 INFO:tasks.ceph.osd.7.smithi176.stderr:2024-03-29T11:20:55.830+0000 7f222a311640 -1 osd.7 0 log_to_monitors true 2024-03-29T11:20:55.838 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-03-29T11:20:55.861+0000 7f487a012640 -1 osd.4 0 log_to_monitors true 2024-03-29T11:20:55.846 INFO:tasks.ceph.osd.1.smithi114.stderr:2024-03-29T11:20:55.869+0000 7efdcc211640 -1 osd.1 0 log_to_monitors true 2024-03-29T11:20:55.851 INFO:tasks.ceph.osd.6.smithi176.stderr:2024-03-29T11:20:55.875+0000 7ff79560c640 -1 osd.6 0 log_to_monitors true 2024-03-29T11:20:55.853 INFO:tasks.ceph.osd.3.smithi114.stderr:2024-03-29T11:20:55.876+0000 7fe5efa0d640 -1 osd.3 0 log_to_monitors true 2024-03-29T11:20:55.871 INFO:tasks.ceph.osd.0.smithi114.stderr:2024-03-29T11:20:55.894+0000 7f5ff939e640 -1 osd.0 0 log_to_monitors true 2024-03-29T11:20:59.711 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:20:59.859 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.882+0000 7f9ace0ba640 10 monclient: get_monmap_and_config 2024-03-29T11:20:59.859 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.882+0000 7f9ace0ba640 10 monclient: build_initial_monmap 2024-03-29T11:20:59.859 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.882+0000 7f9ace0ba640 10 monclient: monmap: 2024-03-29T11:20:59.859 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:20:59.859 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:20:59.859 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:59.883518+0000 2024-03-29T11:20:59.859 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:59.883518+0000 2024-03-29T11:20:59.859 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:59.859 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:20:59.859 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:20:59.860 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T11:20:59.860 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T11:20:59.860 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:20:59.860 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.883+0000 7f9ace0ba640 1 Processor -- start 2024-03-29T11:20:59.860 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.883+0000 7f9ace0ba640 1 -- start start 2024-03-29T11:20:59.860 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.883+0000 7f9ace0ba640 10 monclient: init 2024-03-29T11:20:59.862 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.884+0000 7f9ace0ba640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:59.862 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.884+0000 7f9ace0ba640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:20:59.862 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.885+0000 7f9ace0ba640 10 monclient(hunting): picked mon.noname-a con 0x7f9ac80b0770 addr v1:172.21.15.114:6789/0 2024-03-29T11:20:59.862 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.885+0000 7f9ace0ba640 10 monclient(hunting): picked mon.noname-c con 0x7f9ac814a700 addr v1:172.21.15.176:6789/0 2024-03-29T11:20:59.862 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.885+0000 7f9ace0ba640 10 monclient(hunting): picked mon.noname-b con 0x7f9ac80ac5a0 addr v1:172.21.15.114:6790/0 2024-03-29T11:20:59.862 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.885+0000 7f9ace0ba640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ac80aba70 con 0x7f9ac80b0770 2024-03-29T11:20:59.862 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.885+0000 7f9ace0ba640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ac80aecb0 con 0x7f9ac814a700 2024-03-29T11:20:59.862 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.885+0000 7f9ace0ba640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ac80af150 con 0x7f9ac80ac5a0 2024-03-29T11:20:59.862 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.885+0000 7f9ac6ffd640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f9ac814a700 0x7f9ac80abe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.114:58550/0 (socket says 172.21.15.114:58550) 2024-03-29T11:20:59.862 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.885+0000 7f9ac6ffd640 1 -- 172.21.15.114:0/727109881 learned_addr learned my addr 172.21.15.114:0/727109881 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:20:59.863 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.885+0000 7f9ace0ba640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:59.863 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.885+0000 7f9ace0ba640 10 monclient(hunting): authenticate will time out at 930.067948s 2024-03-29T11:20:59.864 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.886+0000 7f9ac67fc640 1 -- 172.21.15.114:0/727109881 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 53178447 0 0) 0x7f9ac80aecb0 con 0x7f9ac814a700 2024-03-29T11:20:59.864 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.886+0000 7f9ac67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:20:59.864 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.886+0000 7f9ac67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:59.865 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.886+0000 7f9ac67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:59.865 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.886+0000 7f9ac67fc640 10 monclient(hunting): my global_id is 4228 2024-03-29T11:20:59.865 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.886+0000 7f9ac67fc640 1 -- 172.21.15.114:0/727109881 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ab0003520 con 0x7f9ac814a700 2024-03-29T11:20:59.865 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.886+0000 7f9ac67fc640 1 -- 172.21.15.114:0/727109881 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 838053313 0 0) 0x7f9ac80af150 con 0x7f9ac80ac5a0 2024-03-29T11:20:59.865 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.886+0000 7f9ac67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:20:59.865 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.886+0000 7f9ac67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:59.865 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.886+0000 7f9ac67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:59.865 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.886+0000 7f9ac67fc640 10 monclient(hunting): my global_id is 4226 2024-03-29T11:20:59.865 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.886+0000 7f9ac67fc640 1 -- 172.21.15.114:0/727109881 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ac80aecb0 con 0x7f9ac80ac5a0 2024-03-29T11:20:59.865 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.886+0000 7f9ac67fc640 1 -- 172.21.15.114:0/727109881 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 662792721 0 0) 0x7f9ac80aba70 con 0x7f9ac80b0770 2024-03-29T11:20:59.865 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.886+0000 7f9ac67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:20:59.865 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.886+0000 7f9ac67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:59.865 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.886+0000 7f9ac67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:59.865 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.886+0000 7f9ac67fc640 10 monclient(hunting): my global_id is 4236 2024-03-29T11:20:59.866 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.886+0000 7f9ac67fc640 1 -- 172.21.15.114:0/727109881 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ac80af150 con 0x7f9ac80b0770 2024-03-29T11:20:59.866 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.886+0000 7f9ac67fc640 1 -- 172.21.15.114:0/727109881 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1176432755 0 0) 0x7f9ab0003520 con 0x7f9ac814a700 2024-03-29T11:20:59.866 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.886+0000 7f9ac67fc640 1 -- 172.21.15.114:0/727109881 >> v1:172.21.15.114:6790/0 conn(0x7f9ac80ac5a0 legacy=0x7f9ac8151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:59.866 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.886+0000 7f9ac67fc640 1 -- 172.21.15.114:0/727109881 >> v1:172.21.15.114:6789/0 conn(0x7f9ac80b0770 legacy=0x7f9ac8149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:59.866 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.886+0000 7f9ac67fc640 10 monclient: _finish_hunting 0 2024-03-29T11:20:59.866 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.886+0000 7f9ac67fc640 1 monclient: found mon.noname-c 2024-03-29T11:20:59.866 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.886+0000 7f9ac67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:59.866 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.886+0000 7f9ac67fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.176:6789/0 2024-03-29T11:20:59.866 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.887+0000 7f9ac67fc640 1 -- 172.21.15.114:0/727109881 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ac80ac9d0 con 0x7f9ac814a700 2024-03-29T11:20:59.866 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.887+0000 7f9ac67fc640 10 monclient: _finish_auth 0 2024-03-29T11:20:59.866 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.887+0000 7f9ac67fc640 10 monclient: _check_auth_tickets 2024-03-29T11:20:59.866 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.887+0000 7f9ac67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:59.866 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.887+0000 7f9ac67fc640 1 -- 172.21.15.114:0/727109881 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f9abc0016b0 con 0x7f9ac814a700 2024-03-29T11:20:59.866 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.887+0000 7f9ac67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:59.867 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.887+0000 7f9ac67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:20:59.867 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.887+0000 7f9ac67fc640 10 monclient: dump: 2024-03-29T11:20:59.867 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:20:59.867 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:20:59.867 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:59.867 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:59.867 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:59.867 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:20:59.867 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:20:59.867 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:20:59.867 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:20:59.867 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:20:59.867 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.887+0000 7f9ace0ba640 5 monclient: authenticate success, global_id 4228 2024-03-29T11:20:59.867 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.887+0000 7f9ace0ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:59.867 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.887+0000 7f9ac67fc640 1 -- 172.21.15.114:0/727109881 <== mon.1 v1:172.21.15.176:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9abc0018d0 con 0x7f9ac814a700 2024-03-29T11:20:59.867 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.887+0000 7f9ac67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:59.868 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.887+0000 7f9ac67fc640 1 -- 172.21.15.114:0/727109881 <== mon.1 v1:172.21.15.176:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f9abc001c60 con 0x7f9ac814a700 2024-03-29T11:20:59.868 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.887+0000 7f9ac67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:59.868 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.887+0000 7f9ac67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:20:59.868 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.887+0000 7f9ac67fc640 10 monclient: dump: 2024-03-29T11:20:59.868 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:20:59.868 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:20:59.868 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:59.868 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:59.868 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:59.868 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:20:59.868 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:20:59.868 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:20:59.868 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:20:59.868 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:20:59.868 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.887+0000 7f9ace0ba640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:59.869 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.887+0000 7f9ace0ba640 4 set_mon_vals no callback set 2024-03-29T11:20:59.873 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.888+0000 7f9ace0ba640 10 monclient: shutdown 2024-03-29T11:20:59.873 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.888+0000 7f9ace0ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:59.873 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.888+0000 7f9ace0ba640 1 -- 172.21.15.114:0/727109881 >> v1:172.21.15.176:6789/0 conn(0x7f9ac814a700 legacy=0x7f9ac80abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:59.873 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.888+0000 7f9ace0ba640 1 -- 172.21.15.114:0/727109881 shutdown_connections 2024-03-29T11:20:59.873 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.888+0000 7f9ace0ba640 1 -- 172.21.15.114:0/727109881 >> 172.21.15.114:0/727109881 conn(0x7f9ac806b660 msgr2=0x7f9ac806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:59.873 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.889+0000 7f9ace0ba640 1 -- 172.21.15.114:0/727109881 shutdown_connections 2024-03-29T11:20:59.873 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.889+0000 7f9ace0ba640 1 -- 172.21.15.114:0/727109881 wait complete. 2024-03-29T11:20:59.873 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.890+0000 7f9ace0ba640 10 monclient: build_initial_monmap 2024-03-29T11:20:59.873 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.890+0000 7f9ace0ba640 10 monclient: monmap: 2024-03-29T11:20:59.873 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:20:59.874 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:59.874 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T11:20:59.874 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T11:20:59.874 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:59.874 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:20:59.874 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:20:59.874 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T11:20:59.874 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T11:20:59.874 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:20:59.874 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.891+0000 7f9ace0ba640 1 librados: starting msgr at 2024-03-29T11:20:59.874 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.891+0000 7f9ace0ba640 1 librados: starting objecter 2024-03-29T11:20:59.874 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.891+0000 7f9ace0ba640 1 Processor -- start 2024-03-29T11:20:59.874 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.891+0000 7f9ace0ba640 1 -- start start 2024-03-29T11:20:59.874 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.891+0000 7f9ace0ba640 1 librados: setting wanted keys 2024-03-29T11:20:59.874 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.891+0000 7f9ace0ba640 1 librados: calling monclient init 2024-03-29T11:20:59.875 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.891+0000 7f9ace0ba640 10 monclient: init 2024-03-29T11:20:59.875 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.892+0000 7f9ace0ba640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:59.875 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.892+0000 7f9ace0ba640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:20:59.875 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.892+0000 7f9ace0ba640 10 monclient(hunting): picked mon.noname-b con 0x7f9ac80ac5a0 addr v1:172.21.15.176:6789/0 2024-03-29T11:20:59.875 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.892+0000 7f9ace0ba640 10 monclient(hunting): picked mon.noname-c con 0x7f9ac80b0770 addr v1:172.21.15.114:6790/0 2024-03-29T11:20:59.875 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.892+0000 7f9ace0ba640 10 monclient(hunting): picked mon.noname-a con 0x7f9ac814a700 addr v1:172.21.15.114:6789/0 2024-03-29T11:20:59.875 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.893+0000 7f9ace0ba640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ac81dfa80 con 0x7f9ac814a700 2024-03-29T11:20:59.875 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.893+0000 7f9ace0ba640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ac8142080 con 0x7f9ac80ac5a0 2024-03-29T11:20:59.875 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.893+0000 7f9ace0ba640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ac81dff10 con 0x7f9ac80b0770 2024-03-29T11:20:59.875 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.893+0000 7f9ace0ba640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:59.875 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.893+0000 7f9ace0ba640 10 monclient(hunting): authenticate will time out at 930.075180s 2024-03-29T11:20:59.875 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.893+0000 7f9ac6ffd640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f9ac80b0770 0x7f9ac8141260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:44184/0 (socket says 172.21.15.114:44184) 2024-03-29T11:20:59.875 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.893+0000 7f9ac6ffd640 1 -- 172.21.15.114:0/2895117304 learned_addr learned my addr 172.21.15.114:0/2895117304 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:20:59.875 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.893+0000 7f9aaffff640 10 client.?.objecter ms_handle_connect 0x7f9ac814a700 2024-03-29T11:20:59.875 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.893+0000 7f9aaffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:59.876 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.893+0000 7f9aaffff640 10 client.?.objecter ms_handle_connect 0x7f9ac80ac5a0 2024-03-29T11:20:59.876 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.893+0000 7f9aaffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:59.876 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.894+0000 7f9aaffff640 1 -- 172.21.15.114:0/2895117304 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1455331824 0 0) 0x7f9ac8142080 con 0x7f9ac80ac5a0 2024-03-29T11:20:59.876 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.894+0000 7f9aaffff640 10 monclient(hunting): _negotiate 2024-03-29T11:20:59.876 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.894+0000 7f9aaffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:59.876 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.894+0000 7f9aaffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:59.876 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.894+0000 7f9aaffff640 10 monclient(hunting): my global_id is 4231 2024-03-29T11:20:59.876 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.894+0000 7f9aaffff640 1 -- 172.21.15.114:0/2895117304 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ab4004eb0 con 0x7f9ac80ac5a0 2024-03-29T11:20:59.876 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.894+0000 7f9aaffff640 1 -- 172.21.15.114:0/2895117304 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2371780491 0 0) 0x7f9ac81dfa80 con 0x7f9ac814a700 2024-03-29T11:20:59.876 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.894+0000 7f9aaffff640 10 monclient(hunting): _negotiate 2024-03-29T11:20:59.876 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.894+0000 7f9aaffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:59.876 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.894+0000 7f9aaffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:59.876 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.894+0000 7f9aaffff640 10 monclient(hunting): my global_id is 4239 2024-03-29T11:20:59.876 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.894+0000 7f9aaffff640 1 -- 172.21.15.114:0/2895117304 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ac8142080 con 0x7f9ac814a700 2024-03-29T11:20:59.877 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.894+0000 7f9aaffff640 1 -- 172.21.15.114:0/2895117304 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1813790364 0 0) 0x7f9ab4004eb0 con 0x7f9ac80ac5a0 2024-03-29T11:20:59.877 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.894+0000 7f9aaffff640 1 -- 172.21.15.114:0/2895117304 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9ac81dfa80 con 0x7f9ac80ac5a0 2024-03-29T11:20:59.877 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.894+0000 7f9aaffff640 1 -- 172.21.15.114:0/2895117304 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f9ab80018a0 con 0x7f9ac80ac5a0 2024-03-29T11:20:59.877 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.894+0000 7f9aaffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:59.877 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.894+0000 7f9aaffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:20:59.877 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.894+0000 7f9aaffff640 10 monclient(hunting): dump: 2024-03-29T11:20:59.877 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:20:59.877 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:20:59.877 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:59.877 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:59.877 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:59.877 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:20:59.877 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:20:59.877 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:20:59.877 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:20:59.878 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:20:59.878 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.894+0000 7f9aaffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:59.878 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.894+0000 7f9aaffff640 10 client.?.objecter ms_handle_connect 0x7f9ac80b0770 2024-03-29T11:20:59.878 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.894+0000 7f9aaffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:59.878 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.895+0000 7f9aaffff640 1 -- 172.21.15.114:0/2895117304 <== mon.1 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2177300579 0 0) 0x7f9ac81dfa80 con 0x7f9ac80ac5a0 2024-03-29T11:20:59.878 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.895+0000 7f9aaffff640 1 -- 172.21.15.114:0/2895117304 >> v1:172.21.15.114:6790/0 conn(0x7f9ac80b0770 legacy=0x7f9ac8141260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:59.878 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.895+0000 7f9aaffff640 1 -- 172.21.15.114:0/2895117304 >> v1:172.21.15.114:6789/0 conn(0x7f9ac814a700 legacy=0x7f9ac8141970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:59.878 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.895+0000 7f9aaffff640 10 monclient: _finish_hunting 0 2024-03-29T11:20:59.878 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.895+0000 7f9aaffff640 1 monclient: found mon.b 2024-03-29T11:20:59.878 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.895+0000 7f9aaffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:59.878 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.895+0000 7f9aaffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:20:59.878 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.895+0000 7f9aaffff640 1 -- 172.21.15.114:0/2895117304 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ac81240e0 con 0x7f9ac80ac5a0 2024-03-29T11:20:59.878 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.895+0000 7f9aaffff640 10 monclient: _finish_auth 0 2024-03-29T11:20:59.879 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.895+0000 7f9aaffff640 10 monclient: _check_auth_tickets 2024-03-29T11:20:59.879 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.895+0000 7f9aaffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:59.879 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.895+0000 7f9ace0ba640 5 monclient: authenticate success, global_id 4231 2024-03-29T11:20:59.879 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.895+0000 7f9ace0ba640 10 monclient: _renew_subs 2024-03-29T11:20:59.879 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.895+0000 7f9ace0ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:20:59.879 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.895+0000 7f9ace0ba640 1 -- 172.21.15.114:0/2895117304 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ac8124330 con 0x7f9ac80ac5a0 2024-03-29T11:20:59.879 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.895+0000 7f9ace0ba640 10 client.4231.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:59.879 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.895+0000 7f9ace0ba640 10 monclient: _renew_subs 2024-03-29T11:20:59.879 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.895+0000 7f9ace0ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:20:59.879 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.895+0000 7f9ace0ba640 1 -- 172.21.15.114:0/2895117304 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ac81424a0 con 0x7f9ac80ac5a0 2024-03-29T11:20:59.879 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.895+0000 7f9ace0ba640 1 librados: init done 2024-03-29T11:20:59.879 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.895+0000 7f9aaffff640 1 -- 172.21.15.114:0/2895117304 <== mon.1 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9ab8003610 con 0x7f9ac80ac5a0 2024-03-29T11:20:59.879 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.895+0000 7f9aaffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:59.879 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.895+0000 7f9aaffff640 1 -- 172.21.15.114:0/2895117304 <== mon.1 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f9ab80039e0 con 0x7f9ac80ac5a0 2024-03-29T11:20:59.880 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.895+0000 7f9aaffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:59.880 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.895+0000 7f9aaffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:20:59.880 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.895+0000 7f9aaffff640 10 monclient: dump: 2024-03-29T11:20:59.880 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:20:59.880 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:20:59.880 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:59.880 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:20:59.880 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:59.880 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:20:59.880 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:20:59.880 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:20:59.880 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:20:59.880 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:20:59.880 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.895+0000 7f9ac57fa640 4 set_mon_vals no callback set 2024-03-29T11:20:59.881 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.898+0000 7f9aaffff640 1 -- 172.21.15.114:0/2895117304 <== mon.1 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1594292461 0 0) 0x7f9ab8003b40 con 0x7f9ac80ac5a0 2024-03-29T11:20:59.881 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.898+0000 7f9ace0ba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:59.881 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.898+0000 7f9ace0ba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:59.881 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.898+0000 7f9ace0ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:20:59.881 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.898+0000 7f9ace0ba640 1 -- 172.21.15.114:0/2895117304 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9aa8005080 con 0x7f9ac80ac5a0 2024-03-29T11:20:59.882 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.899+0000 7f9ac6ffd640 10 monclient: get_auth_request con 0x7f9ab406de20 auth_method 0 2024-03-29T11:20:59.882 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.899+0000 7f9aaffff640 1 -- 172.21.15.114:0/2895117304 <== mon.1 v1:172.21.15.176:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3238987313 0 0) 0x7f9ab8083880 con 0x7f9ac80ac5a0 2024-03-29T11:20:59.882 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.899+0000 7f9aaffff640 10 client.4231.objecter ms_dispatch 0x7f9ac8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:20:59.882 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.899+0000 7f9aaffff640 3 client.4231.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:20:59.882 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.899+0000 7f9aaffff640 3 client.4231.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:20:59.882 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.899+0000 7f9aaffff640 20 client.4231.objecter dump_active .. 0 homeless 2024-03-29T11:20:59.882 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.899+0000 7f9aaffff640 10 client.4231.objecter ms_handle_connect 0x7f9ab406de20 2024-03-29T11:20:59.883 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.906+0000 7f9aaffff640 1 -- 172.21.15.114:0/2895117304 <== mon.1 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9ab804f160 con 0x7f9ac80ac5a0 2024-03-29T11:20:59.883 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.906+0000 7f9aaffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:59.883 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:20:59.906+0000 7f9aaffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:00.251 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:00.275+0000 7f9ace0ba640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:00.251 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:00.275+0000 7f9ace0ba640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:00.252 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:00.275+0000 7f9ace0ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:21:00.252 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:00.275+0000 7f9ace0ba640 1 -- 172.21.15.114:0/2895117304 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9aa8005330 con 0x7f9ac80ac5a0 2024-03-29T11:21:00.252 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:00.276+0000 7f9aaffff640 1 -- 172.21.15.114:0/2895117304 <== mon.1 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2858224613) 0x7f9ab8051090 con 0x7f9ac80ac5a0 2024-03-29T11:21:00.252 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:00.276+0000 7f9aaffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:00.252 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:00.276+0000 7f9aaffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:00.253 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T11:21:00.253 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"b0de2b9b-c45a-44c7-b6ee-e83b39a1e108","created":"2024-03-29T11:20:49.420660+0000","modified":"2024-03-29T11:20:51.700167+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:51.700167+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a64785d3-6b11-499a-b706-cc827ab96a74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5882dadb-e7a8-4604-ba9a-418013c106c8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d7e8c409-bc66-4e4a-bbdb-420bad31388e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0682989-3595-4cd6-90a5-ab9111c278e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"67068120-e3ff-4817-bd06-44feb7479a35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09e8b-8a85-4997-bdea-75d754e6ee67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2f0d9ead-f47f-4cb3-92a4-0a5ad23cd831","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fdb6a25f-6bd8-4c65-9b9b-b460ba7673a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:00.256 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:00.279+0000 7f9ace0ba640 10 librados: watch_flush enter 2024-03-29T11:21:00.257 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:00.279+0000 7f9ace0ba640 10 librados: watch_flush exit 2024-03-29T11:21:00.257 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:00.279+0000 7f9ace0ba640 20 client.4231.objecter shutdown clearing up homeless session... 2024-03-29T11:21:00.257 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:00.279+0000 7f9ace0ba640 10 client.4231.objecter successfully canceled tick 2024-03-29T11:21:00.257 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:00.279+0000 7f9ace0ba640 1 -- 172.21.15.114:0/2895117304 >> v1:172.21.15.114:6816/4038755055 conn(0x7f9ab406de20 legacy=0x7f9ab40702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:00.257 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:00.279+0000 7f9ace0ba640 10 monclient: shutdown 2024-03-29T11:21:00.257 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:00.279+0000 7f9ace0ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:00.257 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:00.279+0000 7f9ace0ba640 1 -- 172.21.15.114:0/2895117304 >> v1:172.21.15.176:6789/0 conn(0x7f9ac80ac5a0 legacy=0x7f9ac81239d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:00.257 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:00.280+0000 7f9ace0ba640 1 -- 172.21.15.114:0/2895117304 shutdown_connections 2024-03-29T11:21:00.257 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:00.280+0000 7f9ace0ba640 1 -- 172.21.15.114:0/2895117304 >> 172.21.15.114:0/2895117304 conn(0x7f9ac806b660 msgr2=0x7f9ac806ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:00.257 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:00.280+0000 7f9ace0ba640 1 -- 172.21.15.114:0/2895117304 shutdown_connections 2024-03-29T11:21:00.257 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:00.280+0000 7f9ace0ba640 1 -- 172.21.15.114:0/2895117304 wait complete. 2024-03-29T11:21:00.257 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:00.280+0000 7f9ace0ba640 1 librados: shutdown 2024-03-29T11:21:00.269 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:07.075 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:07.221 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.244+0000 7fa268d5a640 10 monclient: get_monmap_and_config 2024-03-29T11:21:07.221 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.244+0000 7fa268d5a640 10 monclient: build_initial_monmap 2024-03-29T11:21:07.221 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.244+0000 7fa268d5a640 10 monclient: monmap: 2024-03-29T11:21:07.221 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:21:07.221 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:21:07.221 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:21:07.245658+0000 2024-03-29T11:21:07.221 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:21:07.245658+0000 2024-03-29T11:21:07.222 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:07.222 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:21:07.222 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:21:07.222 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T11:21:07.222 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T11:21:07.222 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:21:07.222 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.245+0000 7fa268d5a640 1 Processor -- start 2024-03-29T11:21:07.222 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.245+0000 7fa268d5a640 1 -- start start 2024-03-29T11:21:07.223 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.245+0000 7fa268d5a640 10 monclient: init 2024-03-29T11:21:07.223 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.246+0000 7fa268d5a640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:07.223 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.246+0000 7fa268d5a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:21:07.223 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.246+0000 7fa268d5a640 10 monclient(hunting): picked mon.noname-c con 0x7fa2640b2800 addr v1:172.21.15.176:6789/0 2024-03-29T11:21:07.223 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.246+0000 7fa268d5a640 10 monclient(hunting): picked mon.noname-a con 0x7fa2640a9c00 addr v1:172.21.15.114:6789/0 2024-03-29T11:21:07.223 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.246+0000 7fa268d5a640 10 monclient(hunting): picked mon.noname-b con 0x7fa2640b38c0 addr v1:172.21.15.114:6790/0 2024-03-29T11:21:07.223 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.246+0000 7fa268d5a640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa264141eb0 con 0x7fa2640a9c00 2024-03-29T11:21:07.223 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.246+0000 7fa268d5a640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2640b22d0 con 0x7fa2640b2800 2024-03-29T11:21:07.224 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.246+0000 7fa268d5a640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa264142d30 con 0x7fa2640b38c0 2024-03-29T11:21:07.224 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.246+0000 7fa268d5a640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:07.224 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.246+0000 7fa268d5a640 10 monclient(hunting): authenticate will time out at 937.429046s 2024-03-29T11:21:07.224 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.247+0000 7fa262ffd640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fa2640a9c00 0x7fa26413bb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:47234/0 (socket says 172.21.15.114:47234) 2024-03-29T11:21:07.224 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.247+0000 7fa262ffd640 1 -- 172.21.15.114:0/2096646663 learned_addr learned my addr 172.21.15.114:0/2096646663 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:21:07.227 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.247+0000 7fa2627fc640 1 -- 172.21.15.114:0/2096646663 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 312598847 0 0) 0x7fa2640b22d0 con 0x7fa2640b2800 2024-03-29T11:21:07.227 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.247+0000 7fa2627fc640 10 monclient(hunting): _negotiate 2024-03-29T11:21:07.227 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.247+0000 7fa2627fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:07.228 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.247+0000 7fa2627fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:07.228 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.247+0000 7fa2627fc640 10 monclient(hunting): my global_id is 4234 2024-03-29T11:21:07.228 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.247+0000 7fa2627fc640 1 -- 172.21.15.114:0/2096646663 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa24c003520 con 0x7fa2640b2800 2024-03-29T11:21:07.228 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.247+0000 7fa2627fc640 1 -- 172.21.15.114:0/2096646663 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2179446746 0 0) 0x7fa264142d30 con 0x7fa2640b38c0 2024-03-29T11:21:07.228 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.247+0000 7fa2627fc640 10 monclient(hunting): _negotiate 2024-03-29T11:21:07.228 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.247+0000 7fa2627fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:07.228 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.247+0000 7fa2627fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:07.228 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.247+0000 7fa2627fc640 10 monclient(hunting): my global_id is 4232 2024-03-29T11:21:07.228 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.247+0000 7fa2627fc640 1 -- 172.21.15.114:0/2096646663 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2640b22d0 con 0x7fa2640b38c0 2024-03-29T11:21:07.228 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.248+0000 7fa2627fc640 1 -- 172.21.15.114:0/2096646663 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3381588088 0 0) 0x7fa2640b22d0 con 0x7fa2640b38c0 2024-03-29T11:21:07.228 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.248+0000 7fa2627fc640 1 -- 172.21.15.114:0/2096646663 >> v1:172.21.15.176:6789/0 conn(0x7fa2640b2800 legacy=0x7fa264142620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:07.228 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.248+0000 7fa2627fc640 1 -- 172.21.15.114:0/2096646663 >> v1:172.21.15.114:6789/0 conn(0x7fa2640a9c00 legacy=0x7fa26413bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:07.228 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.248+0000 7fa2627fc640 10 monclient: _finish_hunting 0 2024-03-29T11:21:07.228 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.248+0000 7fa2627fc640 1 monclient: found mon.noname-b 2024-03-29T11:21:07.228 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.248+0000 7fa2627fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:07.229 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.248+0000 7fa2627fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.114:6790/0 2024-03-29T11:21:07.229 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.248+0000 7fa2627fc640 1 -- 172.21.15.114:0/2096646663 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2640aaf90 con 0x7fa2640b38c0 2024-03-29T11:21:07.229 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.248+0000 7fa2627fc640 10 monclient: _finish_auth 0 2024-03-29T11:21:07.229 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.248+0000 7fa2627fc640 10 monclient: _check_auth_tickets 2024-03-29T11:21:07.229 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.248+0000 7fa2627fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:07.229 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.248+0000 7fa2627fc640 1 -- 172.21.15.114:0/2096646663 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fa25c0019f0 con 0x7fa2640b38c0 2024-03-29T11:21:07.229 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.248+0000 7fa2627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:07.229 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.248+0000 7fa2627fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:21:07.229 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.248+0000 7fa2627fc640 10 monclient: dump: 2024-03-29T11:21:07.229 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:21:07.229 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:21:07.229 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:21:07.229 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:21:07.229 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:07.229 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:21:07.230 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:21:07.230 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:21:07.230 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:21:07.230 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:21:07.230 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.248+0000 7fa268d5a640 5 monclient: authenticate success, global_id 4232 2024-03-29T11:21:07.230 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.248+0000 7fa268d5a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:07.230 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.248+0000 7fa2627fc640 1 -- 172.21.15.114:0/2096646663 <== mon.2 v1:172.21.15.114:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa25c001b50 con 0x7fa2640b38c0 2024-03-29T11:21:07.230 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.248+0000 7fa2627fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:07.230 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.248+0000 7fa2627fc640 1 -- 172.21.15.114:0/2096646663 <== mon.2 v1:172.21.15.114:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fa25c001ea0 con 0x7fa2640b38c0 2024-03-29T11:21:07.230 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.248+0000 7fa2627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:07.230 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.248+0000 7fa2627fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:21:07.230 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.248+0000 7fa2627fc640 10 monclient: dump: 2024-03-29T11:21:07.230 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:21:07.230 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:21:07.231 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:21:07.231 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:21:07.231 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:07.231 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:21:07.231 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:21:07.231 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:21:07.231 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:21:07.231 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:21:07.231 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.248+0000 7fa268d5a640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:07.231 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.248+0000 7fa268d5a640 4 set_mon_vals no callback set 2024-03-29T11:21:07.231 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.249+0000 7fa268d5a640 10 monclient: shutdown 2024-03-29T11:21:07.231 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.249+0000 7fa268d5a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:07.231 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.249+0000 7fa268d5a640 1 -- 172.21.15.114:0/2096646663 >> v1:172.21.15.114:6790/0 conn(0x7fa2640b38c0 legacy=0x7fa26414d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:07.231 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.249+0000 7fa268d5a640 1 -- 172.21.15.114:0/2096646663 shutdown_connections 2024-03-29T11:21:07.232 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.249+0000 7fa268d5a640 1 -- 172.21.15.114:0/2096646663 >> 172.21.15.114:0/2096646663 conn(0x7fa264066a00 msgr2=0x7fa264066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:07.232 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.249+0000 7fa268d5a640 1 -- 172.21.15.114:0/2096646663 shutdown_connections 2024-03-29T11:21:07.232 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.249+0000 7fa268d5a640 1 -- 172.21.15.114:0/2096646663 wait complete. 2024-03-29T11:21:07.232 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.250+0000 7fa268d5a640 10 monclient: build_initial_monmap 2024-03-29T11:21:07.232 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.250+0000 7fa268d5a640 10 monclient: monmap: 2024-03-29T11:21:07.232 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:21:07.232 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:07.232 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T11:21:07.232 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T11:21:07.232 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:07.232 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:21:07.232 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:21:07.232 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T11:21:07.232 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T11:21:07.232 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:21:07.232 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.251+0000 7fa268d5a640 1 librados: starting msgr at 2024-03-29T11:21:07.233 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.251+0000 7fa268d5a640 1 librados: starting objecter 2024-03-29T11:21:07.233 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.251+0000 7fa268d5a640 1 Processor -- start 2024-03-29T11:21:07.237 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.251+0000 7fa268d5a640 1 -- start start 2024-03-29T11:21:07.237 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.251+0000 7fa268d5a640 1 librados: setting wanted keys 2024-03-29T11:21:07.237 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.251+0000 7fa268d5a640 1 librados: calling monclient init 2024-03-29T11:21:07.237 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.251+0000 7fa268d5a640 10 monclient: init 2024-03-29T11:21:07.237 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.252+0000 7fa268d5a640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:07.237 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.252+0000 7fa268d5a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:21:07.237 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.252+0000 7fa268d5a640 10 monclient(hunting): picked mon.noname-a con 0x7fa2640a9c00 addr v1:172.21.15.114:6789/0 2024-03-29T11:21:07.238 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.252+0000 7fa268d5a640 10 monclient(hunting): picked mon.noname-b con 0x7fa2640b2800 addr v1:172.21.15.176:6789/0 2024-03-29T11:21:07.238 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.252+0000 7fa2637fe640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fa2640a9c00 0x7fa26406fc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:51124/0 (socket says 172.21.15.114:51124) 2024-03-29T11:21:07.238 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.252+0000 7fa2637fe640 1 -- 172.21.15.114:0/29200854 learned_addr learned my addr 172.21.15.114:0/29200854 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:21:07.238 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.252+0000 7fa268d5a640 10 monclient(hunting): picked mon.noname-c con 0x7fa2640b38c0 addr v1:172.21.15.114:6790/0 2024-03-29T11:21:07.238 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.252+0000 7fa24bfff640 10 client.?.objecter ms_handle_connect 0x7fa2640a9c00 2024-03-29T11:21:07.238 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.252+0000 7fa24bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:07.238 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.252+0000 7fa268d5a640 1 -- 172.21.15.114:0/29200854 --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa264070490 con 0x7fa2640a9c00 2024-03-29T11:21:07.238 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.252+0000 7fa268d5a640 1 -- 172.21.15.114:0/29200854 --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa264072020 con 0x7fa2640b2800 2024-03-29T11:21:07.238 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.252+0000 7fa268d5a640 1 -- 172.21.15.114:0/29200854 --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2640710c0 con 0x7fa2640b38c0 2024-03-29T11:21:07.238 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.252+0000 7fa268d5a640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:07.238 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.252+0000 7fa268d5a640 10 monclient(hunting): authenticate will time out at 937.434845s 2024-03-29T11:21:07.238 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.253+0000 7fa24bfff640 10 client.?.objecter ms_handle_connect 0x7fa2640b2800 2024-03-29T11:21:07.238 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.253+0000 7fa24bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:07.238 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.253+0000 7fa24bfff640 1 -- 172.21.15.114:0/29200854 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 177557963 0 0) 0x7fa264070490 con 0x7fa2640a9c00 2024-03-29T11:21:07.239 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.253+0000 7fa24bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:21:07.239 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.253+0000 7fa24bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:07.239 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.253+0000 7fa24bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:07.239 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.253+0000 7fa24bfff640 10 monclient(hunting): my global_id is 4242 2024-03-29T11:21:07.239 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.253+0000 7fa24bfff640 1 -- 172.21.15.114:0/29200854 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa250004eb0 con 0x7fa2640a9c00 2024-03-29T11:21:07.239 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.253+0000 7fa24bfff640 1 -- 172.21.15.114:0/29200854 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3383201547 0 0) 0x7fa250004eb0 con 0x7fa2640a9c00 2024-03-29T11:21:07.239 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.253+0000 7fa24bfff640 1 -- 172.21.15.114:0/29200854 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa264070490 con 0x7fa2640a9c00 2024-03-29T11:21:07.239 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.253+0000 7fa24bfff640 1 -- 172.21.15.114:0/29200854 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fa254001a10 con 0x7fa2640a9c00 2024-03-29T11:21:07.239 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.253+0000 7fa24bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:07.239 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.253+0000 7fa24bfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:21:07.239 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.253+0000 7fa24bfff640 10 monclient(hunting): dump: 2024-03-29T11:21:07.239 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:21:07.239 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:21:07.239 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:21:07.239 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:21:07.240 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:07.240 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:21:07.240 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:21:07.240 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:21:07.240 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:21:07.240 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:21:07.240 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.253+0000 7fa24bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:07.240 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.253+0000 7fa24bfff640 1 -- 172.21.15.114:0/29200854 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4040044859 0 0) 0x7fa264072020 con 0x7fa2640b2800 2024-03-29T11:21:07.240 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.253+0000 7fa24bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:21:07.240 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.253+0000 7fa24bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:07.240 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.253+0000 7fa24bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:07.240 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.253+0000 7fa24bfff640 10 monclient(hunting): my global_id is 4237 2024-03-29T11:21:07.240 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.253+0000 7fa24bfff640 1 -- 172.21.15.114:0/29200854 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa250004eb0 con 0x7fa2640b2800 2024-03-29T11:21:07.240 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.253+0000 7fa24bfff640 1 -- 172.21.15.114:0/29200854 <== mon.0 v1:172.21.15.114:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 171713515 0 0) 0x7fa264070490 con 0x7fa2640a9c00 2024-03-29T11:21:07.241 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.253+0000 7fa24bfff640 1 -- 172.21.15.114:0/29200854 >> v1:172.21.15.114:6790/0 conn(0x7fa2640b38c0 legacy=0x7fa2641e5d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:07.241 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.253+0000 7fa24bfff640 1 -- 172.21.15.114:0/29200854 >> v1:172.21.15.176:6789/0 conn(0x7fa2640b2800 legacy=0x7fa2640709b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:07.241 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.254+0000 7fa24bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:21:07.241 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.254+0000 7fa24bfff640 1 monclient: found mon.a 2024-03-29T11:21:07.241 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.254+0000 7fa24bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:07.241 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.254+0000 7fa24bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T11:21:07.241 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.254+0000 7fa24bfff640 1 -- 172.21.15.114:0/29200854 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa264071250 con 0x7fa2640a9c00 2024-03-29T11:21:07.241 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.254+0000 7fa24bfff640 10 monclient: _finish_auth 0 2024-03-29T11:21:07.241 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.254+0000 7fa24bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:21:07.241 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.254+0000 7fa24bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:07.241 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.254+0000 7fa268d5a640 5 monclient: authenticate success, global_id 4242 2024-03-29T11:21:07.241 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.254+0000 7fa268d5a640 10 monclient: _renew_subs 2024-03-29T11:21:07.241 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.254+0000 7fa268d5a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T11:21:07.241 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.254+0000 7fa268d5a640 1 -- 172.21.15.114:0/29200854 --> v1:172.21.15.114:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2641de820 con 0x7fa2640a9c00 2024-03-29T11:21:07.241 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.254+0000 7fa268d5a640 10 client.4242.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:07.241 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.254+0000 7fa268d5a640 10 monclient: _renew_subs 2024-03-29T11:21:07.242 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.254+0000 7fa268d5a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T11:21:07.242 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.254+0000 7fa268d5a640 1 -- 172.21.15.114:0/29200854 --> v1:172.21.15.114:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa26413ef60 con 0x7fa2640a9c00 2024-03-29T11:21:07.242 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.254+0000 7fa268d5a640 1 librados: init done 2024-03-29T11:21:07.242 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.254+0000 7fa24bfff640 1 -- 172.21.15.114:0/29200854 <== mon.0 v1:172.21.15.114:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa254001d50 con 0x7fa2640a9c00 2024-03-29T11:21:07.242 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.254+0000 7fa24bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:07.242 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.254+0000 7fa24bfff640 1 -- 172.21.15.114:0/29200854 <== mon.0 v1:172.21.15.114:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fa254003a60 con 0x7fa2640a9c00 2024-03-29T11:21:07.242 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.254+0000 7fa24bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:07.242 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.254+0000 7fa24bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:21:07.242 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.254+0000 7fa24bfff640 10 monclient: dump: 2024-03-29T11:21:07.242 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:21:07.242 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:21:07.242 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:21:07.242 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:21:07.243 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:07.243 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:21:07.243 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:21:07.243 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:21:07.243 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:21:07.243 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:21:07.243 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.254+0000 7fa261ffb640 4 set_mon_vals no callback set 2024-03-29T11:21:07.243 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.256+0000 7fa24bfff640 1 -- 172.21.15.114:0/29200854 <== mon.0 v1:172.21.15.114:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1594292461 0 0) 0x7fa254003bc0 con 0x7fa2640a9c00 2024-03-29T11:21:07.243 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.257+0000 7fa24bfff640 1 -- 172.21.15.114:0/29200854 <== mon.0 v1:172.21.15.114:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3238987313 0 0) 0x7fa254084c40 con 0x7fa2640a9c00 2024-03-29T11:21:07.243 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.257+0000 7fa24bfff640 10 client.4242.objecter ms_dispatch 0x7fa264000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:07.243 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.257+0000 7fa24bfff640 3 client.4242.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:07.243 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.257+0000 7fa24bfff640 3 client.4242.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:07.243 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.257+0000 7fa24bfff640 20 client.4242.objecter dump_active .. 0 homeless 2024-03-29T11:21:07.243 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.257+0000 7fa262ffd640 10 monclient: get_auth_request con 0x7fa25006dd60 auth_method 0 2024-03-29T11:21:07.243 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.257+0000 7fa24bfff640 10 client.4242.objecter ms_handle_connect 0x7fa25006dd60 2024-03-29T11:21:07.244 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.257+0000 7fa268d5a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:07.244 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.257+0000 7fa268d5a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:07.244 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.257+0000 7fa268d5a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T11:21:07.244 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.257+0000 7fa268d5a640 1 -- 172.21.15.114:0/29200854 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa244005080 con 0x7fa2640a9c00 2024-03-29T11:21:07.245 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.269+0000 7fa24bfff640 1 -- 172.21.15.114:0/29200854 <== mon.0 v1:172.21.15.114:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa254050350 con 0x7fa2640a9c00 2024-03-29T11:21:07.246 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.269+0000 7fa24bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:07.246 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.269+0000 7fa24bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:07.574 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.597+0000 7fa268d5a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:07.575 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.597+0000 7fa268d5a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:07.575 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.597+0000 7fa268d5a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T11:21:07.575 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.597+0000 7fa268d5a640 1 -- 172.21.15.114:0/29200854 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa244005330 con 0x7fa2640a9c00 2024-03-29T11:21:07.575 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.598+0000 7fa24bfff640 1 -- 172.21.15.114:0/29200854 <== mon.0 v1:172.21.15.114:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2858224613) 0x7fa254052450 con 0x7fa2640a9c00 2024-03-29T11:21:07.575 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.598+0000 7fa24bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:07.575 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.598+0000 7fa24bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:07.575 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T11:21:07.575 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"b0de2b9b-c45a-44c7-b6ee-e83b39a1e108","created":"2024-03-29T11:20:49.420660+0000","modified":"2024-03-29T11:20:51.700167+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:51.700167+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a64785d3-6b11-499a-b706-cc827ab96a74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5882dadb-e7a8-4604-ba9a-418013c106c8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d7e8c409-bc66-4e4a-bbdb-420bad31388e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0682989-3595-4cd6-90a5-ab9111c278e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"67068120-e3ff-4817-bd06-44feb7479a35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09e8b-8a85-4997-bdea-75d754e6ee67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2f0d9ead-f47f-4cb3-92a4-0a5ad23cd831","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fdb6a25f-6bd8-4c65-9b9b-b460ba7673a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:07.578 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.601+0000 7fa268d5a640 10 librados: watch_flush enter 2024-03-29T11:21:07.578 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.601+0000 7fa268d5a640 10 librados: watch_flush exit 2024-03-29T11:21:07.578 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.601+0000 7fa268d5a640 20 client.4242.objecter shutdown clearing up homeless session... 2024-03-29T11:21:07.578 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.601+0000 7fa268d5a640 10 client.4242.objecter successfully canceled tick 2024-03-29T11:21:07.578 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.601+0000 7fa268d5a640 1 -- 172.21.15.114:0/29200854 >> v1:172.21.15.114:6816/4038755055 conn(0x7fa25006dd60 legacy=0x7fa250070220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:07.578 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.601+0000 7fa268d5a640 10 monclient: shutdown 2024-03-29T11:21:07.578 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.601+0000 7fa268d5a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:07.578 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.601+0000 7fa268d5a640 1 -- 172.21.15.114:0/29200854 >> v1:172.21.15.114:6789/0 conn(0x7fa2640a9c00 legacy=0x7fa26406fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:07.578 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.601+0000 7fa268d5a640 1 -- 172.21.15.114:0/29200854 shutdown_connections 2024-03-29T11:21:07.579 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.601+0000 7fa268d5a640 1 -- 172.21.15.114:0/29200854 >> 172.21.15.114:0/29200854 conn(0x7fa264066a00 msgr2=0x7fa2640b3e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:07.579 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.601+0000 7fa268d5a640 1 -- 172.21.15.114:0/29200854 shutdown_connections 2024-03-29T11:21:07.579 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.601+0000 7fa268d5a640 1 -- 172.21.15.114:0/29200854 wait complete. 2024-03-29T11:21:07.579 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:07.601+0000 7fa268d5a640 1 librados: shutdown 2024-03-29T11:21:07.592 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:14.399 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:14.534 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.557+0000 7f10d56e2640 10 monclient: get_monmap_and_config 2024-03-29T11:21:14.534 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.557+0000 7f10d56e2640 10 monclient: build_initial_monmap 2024-03-29T11:21:14.534 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.557+0000 7f10d56e2640 10 monclient: monmap: 2024-03-29T11:21:14.534 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:21:14.534 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:21:14.534 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:21:14.558566+0000 2024-03-29T11:21:14.534 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:21:14.558566+0000 2024-03-29T11:21:14.534 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:14.534 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:21:14.534 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:21:14.534 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T11:21:14.534 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T11:21:14.535 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:21:14.535 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.558+0000 7f10d56e2640 1 Processor -- start 2024-03-29T11:21:14.535 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.558+0000 7f10d56e2640 1 -- start start 2024-03-29T11:21:14.535 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.558+0000 7f10d56e2640 10 monclient: init 2024-03-29T11:21:14.536 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.559+0000 7f10d56e2640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:14.536 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.559+0000 7f10d56e2640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:21:14.536 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.559+0000 7f10d56e2640 10 monclient(hunting): picked mon.noname-c con 0x7f10d01457f0 addr v1:172.21.15.176:6789/0 2024-03-29T11:21:14.536 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.559+0000 7f10d56e2640 10 monclient(hunting): picked mon.noname-b con 0x7f10d013f650 addr v1:172.21.15.114:6790/0 2024-03-29T11:21:14.536 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.559+0000 7f10d56e2640 10 monclient(hunting): picked mon.noname-a con 0x7f10d00a9d70 addr v1:172.21.15.114:6789/0 2024-03-29T11:21:14.536 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.559+0000 7f10d56e2640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10d00aaef0 con 0x7f10d00a9d70 2024-03-29T11:21:14.536 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.559+0000 7f10d56e2640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10d0145bd0 con 0x7f10d01457f0 2024-03-29T11:21:14.536 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.559+0000 7f10d56e2640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10d0145eb0 con 0x7f10d013f650 2024-03-29T11:21:14.536 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.559+0000 7f10d56e2640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:14.536 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.559+0000 7f10d56e2640 10 monclient(hunting): authenticate will time out at 944.741774s 2024-03-29T11:21:14.536 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.559+0000 7f10ceffd640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f10d01457f0 0x7f10d013ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.114:49560/0 (socket says 172.21.15.114:49560) 2024-03-29T11:21:14.537 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.559+0000 7f10ceffd640 1 -- 172.21.15.114:0/1366811660 learned_addr learned my addr 172.21.15.114:0/1366811660 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:21:14.537 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.560+0000 7f10ce7fc640 1 -- 172.21.15.114:0/1366811660 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 116027254 0 0) 0x7f10d0145bd0 con 0x7f10d01457f0 2024-03-29T11:21:14.537 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.560+0000 7f10ce7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:21:14.537 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.560+0000 7f10ce7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:14.537 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.560+0000 7f10ce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:14.537 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.560+0000 7f10ce7fc640 10 monclient(hunting): my global_id is 4240 2024-03-29T11:21:14.538 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.560+0000 7f10ce7fc640 1 -- 172.21.15.114:0/1366811660 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10bc003520 con 0x7f10d01457f0 2024-03-29T11:21:14.538 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.561+0000 7f10ce7fc640 1 -- 172.21.15.114:0/1366811660 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3232563069 0 0) 0x7f10d0145eb0 con 0x7f10d013f650 2024-03-29T11:21:14.538 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.561+0000 7f10ce7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:21:14.538 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.561+0000 7f10ce7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:14.538 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.561+0000 7f10ce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:14.538 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.561+0000 7f10ce7fc640 10 monclient(hunting): my global_id is 4235 2024-03-29T11:21:14.538 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.561+0000 7f10ce7fc640 1 -- 172.21.15.114:0/1366811660 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10d0145bd0 con 0x7f10d013f650 2024-03-29T11:21:14.538 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.561+0000 7f10ce7fc640 1 -- 172.21.15.114:0/1366811660 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3663943069 0 0) 0x7f10d0145bd0 con 0x7f10d013f650 2024-03-29T11:21:14.538 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.561+0000 7f10ce7fc640 1 -- 172.21.15.114:0/1366811660 >> v1:172.21.15.176:6789/0 conn(0x7f10d01457f0 legacy=0x7f10d013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:14.538 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.561+0000 7f10ce7fc640 1 -- 172.21.15.114:0/1366811660 >> v1:172.21.15.114:6789/0 conn(0x7f10d00a9d70 legacy=0x7f10d014d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:21:14.541 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.563+0000 7f10ce7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:21:14.541 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.563+0000 7f10ce7fc640 1 monclient: found mon.noname-b 2024-03-29T11:21:14.541 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.563+0000 7f10ce7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:14.541 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.563+0000 7f10ce7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.114:6790/0 2024-03-29T11:21:14.541 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.563+0000 7f10ce7fc640 1 -- 172.21.15.114:0/1366811660 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10d0146340 con 0x7f10d013f650 2024-03-29T11:21:14.541 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.563+0000 7f10ce7fc640 10 monclient: _finish_auth 0 2024-03-29T11:21:14.542 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.563+0000 7f10ce7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:21:14.542 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.563+0000 7f10ce7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:14.542 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.563+0000 7f10ce7fc640 1 -- 172.21.15.114:0/1366811660 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f10b8001670 con 0x7f10d013f650 2024-03-29T11:21:14.542 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.563+0000 7f10ce7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:14.542 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.563+0000 7f10ce7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:21:14.542 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.563+0000 7f10ce7fc640 10 monclient: dump: 2024-03-29T11:21:14.542 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:21:14.542 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:21:14.542 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:21:14.542 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:21:14.542 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:14.542 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:21:14.542 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:21:14.542 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:21:14.543 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:21:14.543 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:21:14.543 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.563+0000 7f10d56e2640 5 monclient: authenticate success, global_id 4235 2024-03-29T11:21:14.543 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.563+0000 7f10d56e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:14.543 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.564+0000 7f10ce7fc640 1 -- 172.21.15.114:0/1366811660 <== mon.2 v1:172.21.15.114:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f10b80017d0 con 0x7f10d013f650 2024-03-29T11:21:14.543 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.564+0000 7f10ce7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:14.543 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.564+0000 7f10ce7fc640 1 -- 172.21.15.114:0/1366811660 <== mon.2 v1:172.21.15.114:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f10b8001ba0 con 0x7f10d013f650 2024-03-29T11:21:14.543 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.564+0000 7f10ce7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:14.543 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.564+0000 7f10ce7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:21:14.543 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.564+0000 7f10ce7fc640 10 monclient: dump: 2024-03-29T11:21:14.543 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:21:14.543 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:21:14.543 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:21:14.543 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:21:14.543 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:14.543 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:21:14.544 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:21:14.544 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:21:14.544 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:21:14.544 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:21:14.544 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.564+0000 7f10d56e2640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:14.544 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.564+0000 7f10d56e2640 4 set_mon_vals no callback set 2024-03-29T11:21:14.544 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.564+0000 7f10d56e2640 10 monclient: shutdown 2024-03-29T11:21:14.544 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.564+0000 7f10d56e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:14.544 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.564+0000 7f10d56e2640 1 -- 172.21.15.114:0/1366811660 >> v1:172.21.15.114:6790/0 conn(0x7f10d013f650 legacy=0x7f10d00b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:14.544 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.565+0000 7f10d56e2640 1 -- 172.21.15.114:0/1366811660 shutdown_connections 2024-03-29T11:21:14.544 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.565+0000 7f10d56e2640 1 -- 172.21.15.114:0/1366811660 >> 172.21.15.114:0/1366811660 conn(0x7f10d0066a00 msgr2=0x7f10d0066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:14.544 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.565+0000 7f10d56e2640 1 -- 172.21.15.114:0/1366811660 shutdown_connections 2024-03-29T11:21:14.544 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.565+0000 7f10d56e2640 1 -- 172.21.15.114:0/1366811660 wait complete. 2024-03-29T11:21:14.550 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.566+0000 7f10d56e2640 10 monclient: build_initial_monmap 2024-03-29T11:21:14.550 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.566+0000 7f10d56e2640 10 monclient: monmap: 2024-03-29T11:21:14.550 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:21:14.550 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:14.550 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T11:21:14.550 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T11:21:14.550 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:14.550 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:21:14.550 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:21:14.550 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T11:21:14.550 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T11:21:14.550 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:21:14.550 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.566+0000 7f10d56e2640 1 librados: starting msgr at 2024-03-29T11:21:14.550 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.566+0000 7f10d56e2640 1 librados: starting objecter 2024-03-29T11:21:14.551 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.566+0000 7f10d56e2640 1 Processor -- start 2024-03-29T11:21:14.551 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.566+0000 7f10d56e2640 1 -- start start 2024-03-29T11:21:14.551 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.566+0000 7f10d56e2640 1 librados: setting wanted keys 2024-03-29T11:21:14.551 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.566+0000 7f10d56e2640 1 librados: calling monclient init 2024-03-29T11:21:14.551 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.566+0000 7f10d56e2640 10 monclient: init 2024-03-29T11:21:14.551 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.567+0000 7f10d56e2640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:14.551 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.567+0000 7f10d56e2640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:21:14.551 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.567+0000 7f10d56e2640 10 monclient(hunting): picked mon.noname-c con 0x7f10d00a9d70 addr v1:172.21.15.114:6790/0 2024-03-29T11:21:14.551 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.567+0000 7f10d56e2640 10 monclient(hunting): picked mon.noname-b con 0x7f10d013f650 addr v1:172.21.15.176:6789/0 2024-03-29T11:21:14.551 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.568+0000 7f10c7fff640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f10d013f650 0x7f10d01da620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.114:50494/0 (socket says 172.21.15.114:50494) 2024-03-29T11:21:14.551 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.568+0000 7f10ceffd640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f10d00a9d70 0x7f10d00ab870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:34290/0 (socket says 172.21.15.114:34290) 2024-03-29T11:21:14.551 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.568+0000 7f10c7fff640 1 -- 172.21.15.114:0/2652071510 learned_addr learned my addr 172.21.15.114:0/2652071510 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:21:14.551 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.568+0000 7f10c77fe640 10 client.?.objecter ms_handle_connect 0x7f10d00a9d70 2024-03-29T11:21:14.551 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.568+0000 7f10c77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:14.551 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.568+0000 7f10d56e2640 10 monclient(hunting): picked mon.noname-a con 0x7f10d01457f0 addr v1:172.21.15.114:6789/0 2024-03-29T11:21:14.552 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.568+0000 7f10d56e2640 1 -- 172.21.15.114:0/2652071510 --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10d00ace40 con 0x7f10d01457f0 2024-03-29T11:21:14.552 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.568+0000 7f10d56e2640 1 -- 172.21.15.114:0/2652071510 --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10d0142c50 con 0x7f10d013f650 2024-03-29T11:21:14.552 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.568+0000 7f10d56e2640 1 -- 172.21.15.114:0/2652071510 --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10d00abf80 con 0x7f10d00a9d70 2024-03-29T11:21:14.552 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.569+0000 7f10d56e2640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:14.552 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.569+0000 7f10c77fe640 1 -- 172.21.15.114:0/2652071510 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1176710530 0 0) 0x7f10d00abf80 con 0x7f10d00a9d70 2024-03-29T11:21:14.552 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.569+0000 7f10d56e2640 10 monclient(hunting): authenticate will time out at 944.751206s 2024-03-29T11:21:14.552 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.569+0000 7f10c77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:21:14.552 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.569+0000 7f10c77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:14.552 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.569+0000 7f10c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:14.552 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.569+0000 7f10c77fe640 10 monclient(hunting): my global_id is 4238 2024-03-29T11:21:14.552 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.569+0000 7f10c77fe640 1 -- 172.21.15.114:0/2652071510 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10b0004eb0 con 0x7f10d00a9d70 2024-03-29T11:21:14.552 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.569+0000 7f10c77fe640 1 -- 172.21.15.114:0/2652071510 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2995865363 0 0) 0x7f10b0004eb0 con 0x7f10d00a9d70 2024-03-29T11:21:14.552 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.569+0000 7f10c77fe640 1 -- 172.21.15.114:0/2652071510 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f10d00abf80 con 0x7f10d00a9d70 2024-03-29T11:21:14.552 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.569+0000 7f10c77fe640 1 -- 172.21.15.114:0/2652071510 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f10c0001770 con 0x7f10d00a9d70 2024-03-29T11:21:14.553 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.569+0000 7f10c77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:14.553 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.569+0000 7f10c77fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:21:14.553 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.569+0000 7f10c77fe640 10 monclient(hunting): dump: 2024-03-29T11:21:14.553 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:21:14.553 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:21:14.553 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:21:14.553 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:21:14.553 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:14.553 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:21:14.553 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:21:14.553 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:21:14.553 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:21:14.553 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:21:14.553 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.569+0000 7f10c77fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:14.553 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.569+0000 7f10c77fe640 10 client.?.objecter ms_handle_connect 0x7f10d013f650 2024-03-29T11:21:14.554 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.569+0000 7f10c77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:14.554 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.569+0000 7f10c77fe640 1 -- 172.21.15.114:0/2652071510 <== mon.2 v1:172.21.15.114:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2884760357 0 0) 0x7f10d00abf80 con 0x7f10d00a9d70 2024-03-29T11:21:14.554 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.569+0000 7f10c77fe640 1 -- 172.21.15.114:0/2652071510 >> v1:172.21.15.176:6789/0 conn(0x7f10d013f650 legacy=0x7f10d01da620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:14.554 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.569+0000 7f10c77fe640 1 -- 172.21.15.114:0/2652071510 >> v1:172.21.15.114:6789/0 conn(0x7f10d01457f0 legacy=0x7f10d01dad30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:21:14.554 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.570+0000 7f10c77fe640 10 monclient: _finish_hunting 0 2024-03-29T11:21:14.554 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.570+0000 7f10c77fe640 1 monclient: found mon.c 2024-03-29T11:21:14.554 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.570+0000 7f10c77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:14.554 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.570+0000 7f10c77fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:21:14.554 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.570+0000 7f10c77fe640 1 -- 172.21.15.114:0/2652071510 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10d00b1620 con 0x7f10d00a9d70 2024-03-29T11:21:14.554 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.570+0000 7f10c77fe640 10 monclient: _finish_auth 0 2024-03-29T11:21:14.554 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.570+0000 7f10c77fe640 10 monclient: _check_auth_tickets 2024-03-29T11:21:14.554 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.570+0000 7f10c77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:14.554 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.570+0000 7f10d56e2640 5 monclient: authenticate success, global_id 4238 2024-03-29T11:21:14.554 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.570+0000 7f10d56e2640 10 monclient: _renew_subs 2024-03-29T11:21:14.554 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.570+0000 7f10d56e2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:21:14.555 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.570+0000 7f10d56e2640 1 -- 172.21.15.114:0/2652071510 --> v1:172.21.15.114:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10d0142890 con 0x7f10d00a9d70 2024-03-29T11:21:14.555 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.570+0000 7f10d56e2640 10 client.4238.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:14.555 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.570+0000 7f10d56e2640 10 monclient: _renew_subs 2024-03-29T11:21:14.555 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.570+0000 7f10d56e2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:21:14.555 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.570+0000 7f10d56e2640 1 -- 172.21.15.114:0/2652071510 --> v1:172.21.15.114:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f10d01e2c00 con 0x7f10d00a9d70 2024-03-29T11:21:14.555 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.570+0000 7f10d56e2640 1 librados: init done 2024-03-29T11:21:14.555 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.570+0000 7f10c77fe640 1 -- 172.21.15.114:0/2652071510 <== mon.2 v1:172.21.15.114:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f10c0001ce0 con 0x7f10d00a9d70 2024-03-29T11:21:14.555 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.570+0000 7f10c77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:14.555 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.570+0000 7f10c77fe640 1 -- 172.21.15.114:0/2652071510 <== mon.2 v1:172.21.15.114:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f10c0004980 con 0x7f10d00a9d70 2024-03-29T11:21:14.555 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.570+0000 7f10c77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:14.555 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.570+0000 7f10c77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:21:14.555 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.570+0000 7f10c77fe640 10 monclient: dump: 2024-03-29T11:21:14.555 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:21:14.555 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:21:14.556 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:21:14.556 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:21:14.556 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:14.556 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:21:14.556 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:21:14.556 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:21:14.556 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:21:14.556 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:21:14.556 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.570+0000 7f10cdffb640 4 set_mon_vals no callback set 2024-03-29T11:21:14.556 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.572+0000 7f10c77fe640 1 -- 172.21.15.114:0/2652071510 <== mon.2 v1:172.21.15.114:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1594292461 0 0) 0x7f10c0004ae0 con 0x7f10d00a9d70 2024-03-29T11:21:14.556 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.572+0000 7f10d56e2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:14.556 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.572+0000 7f10d56e2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:14.556 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.572+0000 7f10d56e2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:21:14.556 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.572+0000 7f10d56e2640 1 -- 172.21.15.114:0/2652071510 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10b4005080 con 0x7f10d00a9d70 2024-03-29T11:21:14.556 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.573+0000 7f10c77fe640 1 -- 172.21.15.114:0/2652071510 <== mon.2 v1:172.21.15.114:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3238987313 0 0) 0x7f10c00849e0 con 0x7f10d00a9d70 2024-03-29T11:21:14.557 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.573+0000 7f10c77fe640 10 client.4238.objecter ms_dispatch 0x7f10d0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:14.557 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.573+0000 7f10c77fe640 3 client.4238.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:14.557 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.573+0000 7f10c77fe640 3 client.4238.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:14.557 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.573+0000 7f10c77fe640 20 client.4238.objecter dump_active .. 0 homeless 2024-03-29T11:21:14.557 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.573+0000 7f10c7fff640 10 monclient: get_auth_request con 0x7f10b006db20 auth_method 0 2024-03-29T11:21:14.557 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.573+0000 7f10c77fe640 10 client.4238.objecter ms_handle_connect 0x7f10b006db20 2024-03-29T11:21:14.557 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.581+0000 7f10c77fe640 1 -- 172.21.15.114:0/2652071510 <== mon.2 v1:172.21.15.114:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f10c0050320 con 0x7f10d00a9d70 2024-03-29T11:21:14.558 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.581+0000 7f10c77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:14.558 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.581+0000 7f10c77fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:14.872 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.896+0000 7f10d56e2640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:14.872 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.896+0000 7f10d56e2640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:14.873 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.896+0000 7f10d56e2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:21:14.873 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.896+0000 7f10d56e2640 1 -- 172.21.15.114:0/2652071510 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f10b4005330 con 0x7f10d00a9d70 2024-03-29T11:21:14.873 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.897+0000 7f10c77fe640 1 -- 172.21.15.114:0/2652071510 <== mon.2 v1:172.21.15.114:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2858224613) 0x7f10c0052250 con 0x7f10d00a9d70 2024-03-29T11:21:14.873 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.897+0000 7f10c77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:14.873 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.897+0000 7f10c77fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:14.874 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T11:21:14.874 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"b0de2b9b-c45a-44c7-b6ee-e83b39a1e108","created":"2024-03-29T11:20:49.420660+0000","modified":"2024-03-29T11:20:51.700167+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:51.700167+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a64785d3-6b11-499a-b706-cc827ab96a74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5882dadb-e7a8-4604-ba9a-418013c106c8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d7e8c409-bc66-4e4a-bbdb-420bad31388e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0682989-3595-4cd6-90a5-ab9111c278e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"67068120-e3ff-4817-bd06-44feb7479a35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09e8b-8a85-4997-bdea-75d754e6ee67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2f0d9ead-f47f-4cb3-92a4-0a5ad23cd831","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fdb6a25f-6bd8-4c65-9b9b-b460ba7673a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:14.877 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.900+0000 7f10d56e2640 10 librados: watch_flush enter 2024-03-29T11:21:14.878 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.900+0000 7f10d56e2640 10 librados: watch_flush exit 2024-03-29T11:21:14.878 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.900+0000 7f10d56e2640 20 client.4238.objecter shutdown clearing up homeless session... 2024-03-29T11:21:14.878 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.900+0000 7f10d56e2640 10 client.4238.objecter successfully canceled tick 2024-03-29T11:21:14.878 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.901+0000 7f10d56e2640 1 -- 172.21.15.114:0/2652071510 >> v1:172.21.15.114:6816/4038755055 conn(0x7f10b006db20 legacy=0x7f10b006ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:14.878 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.901+0000 7f10d56e2640 10 monclient: shutdown 2024-03-29T11:21:14.878 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.901+0000 7f10d56e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:14.878 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.901+0000 7f10d56e2640 1 -- 172.21.15.114:0/2652071510 >> v1:172.21.15.114:6790/0 conn(0x7f10d00a9d70 legacy=0x7f10d00ab870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:14.878 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.901+0000 7f10d56e2640 1 -- 172.21.15.114:0/2652071510 shutdown_connections 2024-03-29T11:21:14.878 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.901+0000 7f10d56e2640 1 -- 172.21.15.114:0/2652071510 >> 172.21.15.114:0/2652071510 conn(0x7f10d0066a00 msgr2=0x7f10d00b64a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:14.879 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.901+0000 7f10d56e2640 1 -- 172.21.15.114:0/2652071510 shutdown_connections 2024-03-29T11:21:14.879 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.901+0000 7f10d56e2640 1 -- 172.21.15.114:0/2652071510 wait complete. 2024-03-29T11:21:14.879 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:14.902+0000 7f10d56e2640 1 librados: shutdown 2024-03-29T11:21:14.896 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:21.702 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:21.849 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.872+0000 7f8053cf6640 10 monclient: get_monmap_and_config 2024-03-29T11:21:21.849 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.872+0000 7f8053cf6640 10 monclient: build_initial_monmap 2024-03-29T11:21:21.849 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.872+0000 7f8053cf6640 10 monclient: monmap: 2024-03-29T11:21:21.849 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:21:21.850 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:21:21.850 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:21:21.873583+0000 2024-03-29T11:21:21.850 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:21:21.873583+0000 2024-03-29T11:21:21.850 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:21.850 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:21:21.850 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:21:21.850 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T11:21:21.850 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T11:21:21.850 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:21:21.854 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.875+0000 7f8053cf6640 1 Processor -- start 2024-03-29T11:21:21.854 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.875+0000 7f8053cf6640 1 -- start start 2024-03-29T11:21:21.854 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.875+0000 7f8053cf6640 10 monclient: init 2024-03-29T11:21:21.854 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.876+0000 7f8053cf6640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:21.854 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.876+0000 7f8053cf6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:21:21.854 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.876+0000 7f8053cf6640 10 monclient(hunting): picked mon.noname-a con 0x7f804c1422b0 addr v1:172.21.15.114:6789/0 2024-03-29T11:21:21.854 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.876+0000 7f8053cf6640 10 monclient(hunting): picked mon.noname-b con 0x7f804c142690 addr v1:172.21.15.114:6790/0 2024-03-29T11:21:21.854 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.876+0000 7f8053cf6640 10 monclient(hunting): picked mon.noname-c con 0x7f804c0b17d0 addr v1:172.21.15.176:6789/0 2024-03-29T11:21:21.854 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.876+0000 7f8053cf6640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f804c142c80 con 0x7f804c1422b0 2024-03-29T11:21:21.854 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.876+0000 7f8053cf6640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f804c0aae40 con 0x7f804c0b17d0 2024-03-29T11:21:21.854 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.876+0000 7f8053cf6640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f804c13fbd0 con 0x7f804c142690 2024-03-29T11:21:21.855 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.876+0000 7f8053cf6640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:21.855 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.876+0000 7f8053cf6640 10 monclient(hunting): authenticate will time out at 952.058963s 2024-03-29T11:21:21.857 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.880+0000 7f8051291640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f804c142690 0x7f804c0b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:42885/0 (socket says 172.21.15.114:42885) 2024-03-29T11:21:21.857 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.880+0000 7f8051291640 1 -- 172.21.15.114:0/3038982978 learned_addr learned my addr 172.21.15.114:0/3038982978 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:21:21.860 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.884+0000 7f8050a90640 1 -- 172.21.15.114:0/3038982978 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3637832464 0 0) 0x7f804c0aae40 con 0x7f804c0b17d0 2024-03-29T11:21:21.860 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.884+0000 7f8050a90640 10 monclient(hunting): _negotiate 2024-03-29T11:21:21.860 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.884+0000 7f8050a90640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:21.860 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.884+0000 7f8050a90640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:21.860 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.884+0000 7f8050a90640 10 monclient(hunting): my global_id is 4246 2024-03-29T11:21:21.861 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.884+0000 7f8050a90640 1 -- 172.21.15.114:0/3038982978 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8038003520 con 0x7f804c0b17d0 2024-03-29T11:21:21.861 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.885+0000 7f8050a90640 1 -- 172.21.15.114:0/3038982978 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3784538808 0 0) 0x7f8038003520 con 0x7f804c0b17d0 2024-03-29T11:21:21.861 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.885+0000 7f8050a90640 1 -- 172.21.15.114:0/3038982978 >> v1:172.21.15.114:6790/0 conn(0x7f804c142690 legacy=0x7f804c0b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:21.861 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.885+0000 7f8050a90640 1 -- 172.21.15.114:0/3038982978 >> v1:172.21.15.114:6789/0 conn(0x7f804c1422b0 legacy=0x7f804c13f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:21:21.861 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.885+0000 7f8050a90640 10 monclient: _finish_hunting 0 2024-03-29T11:21:21.861 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.885+0000 7f8050a90640 1 monclient: found mon.noname-c 2024-03-29T11:21:21.861 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.885+0000 7f8050a90640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:21.862 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.885+0000 7f8050a90640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.176:6789/0 2024-03-29T11:21:21.862 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.885+0000 7f8050a90640 1 -- 172.21.15.114:0/3038982978 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f804c151940 con 0x7f804c0b17d0 2024-03-29T11:21:21.862 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.885+0000 7f8050a90640 10 monclient: _finish_auth 0 2024-03-29T11:21:21.862 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.885+0000 7f8050a90640 10 monclient: _check_auth_tickets 2024-03-29T11:21:21.862 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.885+0000 7f8050a90640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:21.862 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.885+0000 7f8050a90640 1 -- 172.21.15.114:0/3038982978 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f8044001a10 con 0x7f804c0b17d0 2024-03-29T11:21:21.862 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.885+0000 7f8050a90640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:21.862 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.885+0000 7f8050a90640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:21:21.862 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.885+0000 7f8050a90640 10 monclient: dump: 2024-03-29T11:21:21.862 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:21:21.862 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:21:21.862 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:21:21.862 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:21:21.863 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:21.863 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:21:21.863 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:21:21.863 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:21:21.863 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:21:21.863 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:21:21.863 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.885+0000 7f8053cf6640 5 monclient: authenticate success, global_id 4246 2024-03-29T11:21:21.863 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.885+0000 7f8053cf6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:21.864 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.887+0000 7f8050a90640 1 -- 172.21.15.114:0/3038982978 <== mon.1 v1:172.21.15.176:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8044001b70 con 0x7f804c0b17d0 2024-03-29T11:21:21.864 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.887+0000 7f8050a90640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:21.864 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.887+0000 7f8050a90640 1 -- 172.21.15.114:0/3038982978 <== mon.1 v1:172.21.15.176:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f8044001f80 con 0x7f804c0b17d0 2024-03-29T11:21:21.864 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.887+0000 7f8050a90640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:21.864 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.887+0000 7f8050a90640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:21:21.864 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.887+0000 7f8050a90640 10 monclient: dump: 2024-03-29T11:21:21.864 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:21:21.864 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:21:21.864 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:21:21.864 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:21:21.864 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:21.865 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:21:21.865 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:21:21.865 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:21:21.865 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:21:21.865 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:21:21.867 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.890+0000 7f8053cf6640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:21.867 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.890+0000 7f8053cf6640 4 set_mon_vals no callback set 2024-03-29T11:21:21.868 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.891+0000 7f8053cf6640 10 monclient: shutdown 2024-03-29T11:21:21.868 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.891+0000 7f8053cf6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:21.868 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.891+0000 7f8053cf6640 1 -- 172.21.15.114:0/3038982978 >> v1:172.21.15.176:6789/0 conn(0x7f804c0b17d0 legacy=0x7f804c0b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:21.870 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.892+0000 7f8053cf6640 1 -- 172.21.15.114:0/3038982978 shutdown_connections 2024-03-29T11:21:21.870 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.892+0000 7f8053cf6640 1 -- 172.21.15.114:0/3038982978 >> 172.21.15.114:0/3038982978 conn(0x7f804c066a00 msgr2=0x7f804c066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:21.872 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.895+0000 7f8053cf6640 1 -- 172.21.15.114:0/3038982978 shutdown_connections 2024-03-29T11:21:21.872 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.895+0000 7f8053cf6640 1 -- 172.21.15.114:0/3038982978 wait complete. 2024-03-29T11:21:21.874 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.896+0000 7f8053cf6640 10 monclient: build_initial_monmap 2024-03-29T11:21:21.874 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.896+0000 7f8053cf6640 10 monclient: monmap: 2024-03-29T11:21:21.874 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:21:21.874 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:21.874 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T11:21:21.874 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T11:21:21.874 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:21.874 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:21:21.875 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:21:21.875 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T11:21:21.875 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T11:21:21.875 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:21:21.875 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.897+0000 7f8053cf6640 1 librados: starting msgr at 2024-03-29T11:21:21.875 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.897+0000 7f8053cf6640 1 librados: starting objecter 2024-03-29T11:21:21.875 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.897+0000 7f8053cf6640 1 Processor -- start 2024-03-29T11:21:21.875 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.897+0000 7f8053cf6640 1 -- start start 2024-03-29T11:21:21.875 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.897+0000 7f8053cf6640 1 librados: setting wanted keys 2024-03-29T11:21:21.875 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.897+0000 7f8053cf6640 1 librados: calling monclient init 2024-03-29T11:21:21.875 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.897+0000 7f8053cf6640 10 monclient: init 2024-03-29T11:21:21.876 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.898+0000 7f8053cf6640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:21.876 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.898+0000 7f8053cf6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:21:21.876 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.898+0000 7f8053cf6640 10 monclient(hunting): picked mon.noname-b con 0x7f804c0b17d0 addr v1:172.21.15.176:6789/0 2024-03-29T11:21:21.884 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.900+0000 7f8053cf6640 10 monclient(hunting): picked mon.noname-c con 0x7f804c1422b0 addr v1:172.21.15.114:6790/0 2024-03-29T11:21:21.884 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.900+0000 7f8053cf6640 10 monclient(hunting): picked mon.noname-a con 0x7f804c142690 addr v1:172.21.15.114:6789/0 2024-03-29T11:21:21.884 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.900+0000 7f8053cf6640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f804c12fcc0 con 0x7f804c142690 2024-03-29T11:21:21.884 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.900+0000 7f8053cf6640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f804c0ac730 con 0x7f804c0b17d0 2024-03-29T11:21:21.884 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.900+0000 7f8053cf6640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f804c0aca10 con 0x7f804c1422b0 2024-03-29T11:21:21.884 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.900+0000 7f8053cf6640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:21.884 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.900+0000 7f8053cf6640 10 monclient(hunting): authenticate will time out at 952.082799s 2024-03-29T11:21:21.884 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.904+0000 7f8052293640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f804c142690 0x7f804c1e1d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:57979/0 (socket says 172.21.15.114:57979) 2024-03-29T11:21:21.884 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.904+0000 7f8052293640 1 -- 172.21.15.114:0/3657765178 learned_addr learned my addr 172.21.15.114:0/3657765178 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:21:21.884 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.904+0000 7f80427fc640 10 client.?.objecter ms_handle_connect 0x7f804c142690 2024-03-29T11:21:21.885 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.904+0000 7f80427fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:21.885 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.904+0000 7f80427fc640 10 client.?.objecter ms_handle_connect 0x7f804c1422b0 2024-03-29T11:21:21.885 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.904+0000 7f80427fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:21.885 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.904+0000 7f80427fc640 1 -- 172.21.15.114:0/3657765178 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 525080650 0 0) 0x7f804c12fcc0 con 0x7f804c142690 2024-03-29T11:21:21.885 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.904+0000 7f80427fc640 10 monclient(hunting): _negotiate 2024-03-29T11:21:21.885 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.904+0000 7f80427fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:21.885 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.904+0000 7f80427fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:21.885 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.904+0000 7f80427fc640 10 monclient(hunting): my global_id is 4248 2024-03-29T11:21:21.885 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.904+0000 7f80427fc640 1 -- 172.21.15.114:0/3657765178 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8030004eb0 con 0x7f804c142690 2024-03-29T11:21:21.885 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.905+0000 7f80427fc640 1 -- 172.21.15.114:0/3657765178 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1614471489 0 0) 0x7f8030004eb0 con 0x7f804c142690 2024-03-29T11:21:21.885 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.905+0000 7f80427fc640 1 -- 172.21.15.114:0/3657765178 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f804c12fcc0 con 0x7f804c142690 2024-03-29T11:21:21.885 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.905+0000 7f80427fc640 1 -- 172.21.15.114:0/3657765178 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f8044001f40 con 0x7f804c142690 2024-03-29T11:21:21.885 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.905+0000 7f80427fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:21.885 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.905+0000 7f80427fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:21:21.886 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.905+0000 7f80427fc640 10 monclient(hunting): dump: 2024-03-29T11:21:21.886 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:21:21.886 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:21:21.886 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:21:21.886 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:21:21.886 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:21.886 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:21:21.886 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:21:21.886 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:21:21.886 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:21:21.886 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:21:21.886 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.905+0000 7f80427fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:21.886 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.905+0000 7f80427fc640 1 -- 172.21.15.114:0/3657765178 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3537349766 0 0) 0x7f804c0aca10 con 0x7f804c1422b0 2024-03-29T11:21:21.886 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.905+0000 7f80427fc640 10 monclient(hunting): _negotiate 2024-03-29T11:21:21.886 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.905+0000 7f80427fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:21.887 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.905+0000 7f80427fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:21.887 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.905+0000 7f80427fc640 10 monclient(hunting): my global_id is 4244 2024-03-29T11:21:21.887 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.905+0000 7f80427fc640 1 -- 172.21.15.114:0/3657765178 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8030004eb0 con 0x7f804c1422b0 2024-03-29T11:21:21.887 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.905+0000 7f80427fc640 1 -- 172.21.15.114:0/3657765178 <== mon.0 v1:172.21.15.114:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1841136890 0 0) 0x7f804c12fcc0 con 0x7f804c142690 2024-03-29T11:21:21.887 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.905+0000 7f80427fc640 1 -- 172.21.15.114:0/3657765178 >> v1:172.21.15.114:6790/0 conn(0x7f804c1422b0 legacy=0x7f804c130860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:21.887 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.905+0000 7f80427fc640 1 -- 172.21.15.114:0/3657765178 >> v1:172.21.15.176:6789/0 conn(0x7f804c0b17d0 legacy=0x7f804c130150 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:21:21.887 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.906+0000 7f80427fc640 10 monclient: _finish_hunting 0 2024-03-29T11:21:21.887 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.906+0000 7f80427fc640 1 monclient: found mon.a 2024-03-29T11:21:21.887 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.906+0000 7f80427fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:21.887 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.906+0000 7f80427fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T11:21:21.887 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.906+0000 7f80427fc640 1 -- 172.21.15.114:0/3657765178 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f804c0ab7b0 con 0x7f804c142690 2024-03-29T11:21:21.887 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.906+0000 7f80427fc640 10 monclient: _finish_auth 0 2024-03-29T11:21:21.887 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.906+0000 7f80427fc640 10 monclient: _check_auth_tickets 2024-03-29T11:21:21.887 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.906+0000 7f80427fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:21.887 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.907+0000 7f8053cf6640 5 monclient: authenticate success, global_id 4248 2024-03-29T11:21:21.888 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.907+0000 7f8053cf6640 10 monclient: _renew_subs 2024-03-29T11:21:21.888 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.907+0000 7f8053cf6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T11:21:21.888 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.907+0000 7f8053cf6640 1 -- 172.21.15.114:0/3657765178 --> v1:172.21.15.114:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f804c0aba00 con 0x7f804c142690 2024-03-29T11:21:21.888 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.907+0000 7f8053cf6640 10 client.4248.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:21.888 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.907+0000 7f8053cf6640 10 monclient: _renew_subs 2024-03-29T11:21:21.888 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.907+0000 7f8053cf6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T11:21:21.888 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.907+0000 7f8053cf6640 1 -- 172.21.15.114:0/3657765178 --> v1:172.21.15.114:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f804c1327d0 con 0x7f804c142690 2024-03-29T11:21:21.888 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.907+0000 7f8053cf6640 1 librados: init done 2024-03-29T11:21:21.888 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.907+0000 7f80427fc640 1 -- 172.21.15.114:0/3657765178 <== mon.0 v1:172.21.15.114:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8044003c20 con 0x7f804c142690 2024-03-29T11:21:21.888 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.907+0000 7f80427fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:21.890 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.907+0000 7f80427fc640 1 -- 172.21.15.114:0/3657765178 <== mon.0 v1:172.21.15.114:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f8044004070 con 0x7f804c142690 2024-03-29T11:21:21.890 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.908+0000 7f80427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:21.890 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.908+0000 7f80427fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:21:21.890 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.908+0000 7f80427fc640 10 monclient: dump: 2024-03-29T11:21:21.890 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:21:21.890 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:21:21.890 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:21:21.890 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:21:21.890 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:21.891 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:21:21.891 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:21:21.891 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:21:21.891 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:21:21.891 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:21:21.891 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.907+0000 7f8043fff640 4 set_mon_vals no callback set 2024-03-29T11:21:21.891 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.910+0000 7f80427fc640 1 -- 172.21.15.114:0/3657765178 <== mon.0 v1:172.21.15.114:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1594292461 0 0) 0x7f80440041d0 con 0x7f804c142690 2024-03-29T11:21:21.891 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.910+0000 7f8053cf6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:21.891 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.910+0000 7f8053cf6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:21.891 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.910+0000 7f8053cf6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T11:21:21.891 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.910+0000 7f8053cf6640 1 -- 172.21.15.114:0/3657765178 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f804c075f10 con 0x7f804c142690 2024-03-29T11:21:21.891 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.911+0000 7f80427fc640 1 -- 172.21.15.114:0/3657765178 <== mon.0 v1:172.21.15.114:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3238987313 0 0) 0x7f8044084f00 con 0x7f804c142690 2024-03-29T11:21:21.891 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.911+0000 7f80427fc640 10 client.4248.objecter ms_dispatch 0x7f804c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:21.891 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.911+0000 7f80427fc640 3 client.4248.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:21.891 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.911+0000 7f80427fc640 3 client.4248.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:21.892 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.911+0000 7f80427fc640 20 client.4248.objecter dump_active .. 0 homeless 2024-03-29T11:21:21.892 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.911+0000 7f8051a92640 10 monclient: get_auth_request con 0x7f803006dd60 auth_method 0 2024-03-29T11:21:21.892 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.912+0000 7f80427fc640 10 client.4248.objecter ms_handle_connect 0x7f803006dd60 2024-03-29T11:21:21.900 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.923+0000 7f80427fc640 1 -- 172.21.15.114:0/3657765178 <== mon.0 v1:172.21.15.114:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f80440507e0 con 0x7f804c142690 2024-03-29T11:21:21.900 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.923+0000 7f80427fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:21.900 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:21.923+0000 7f80427fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:22.219 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:22.242+0000 7f8053cf6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:22.219 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:22.243+0000 7f8053cf6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:22.220 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:22.243+0000 7f8053cf6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T11:21:22.220 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:22.243+0000 7f8053cf6640 1 -- 172.21.15.114:0/3657765178 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f804c12fe70 con 0x7f804c142690 2024-03-29T11:21:22.220 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:22.244+0000 7f80427fc640 1 -- 172.21.15.114:0/3657765178 <== mon.0 v1:172.21.15.114:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2858224613) 0x7f804c12fe70 con 0x7f804c142690 2024-03-29T11:21:22.220 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:22.244+0000 7f80427fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:22.221 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T11:21:22.221 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:22.244+0000 7f80427fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:22.221 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"b0de2b9b-c45a-44c7-b6ee-e83b39a1e108","created":"2024-03-29T11:20:49.420660+0000","modified":"2024-03-29T11:20:51.700167+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:51.700167+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a64785d3-6b11-499a-b706-cc827ab96a74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5882dadb-e7a8-4604-ba9a-418013c106c8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d7e8c409-bc66-4e4a-bbdb-420bad31388e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0682989-3595-4cd6-90a5-ab9111c278e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"67068120-e3ff-4817-bd06-44feb7479a35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09e8b-8a85-4997-bdea-75d754e6ee67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2f0d9ead-f47f-4cb3-92a4-0a5ad23cd831","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fdb6a25f-6bd8-4c65-9b9b-b460ba7673a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:22.224 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:22.247+0000 7f8053cf6640 10 librados: watch_flush enter 2024-03-29T11:21:22.224 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:22.248+0000 7f8053cf6640 10 librados: watch_flush exit 2024-03-29T11:21:22.224 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:22.248+0000 7f8053cf6640 20 client.4248.objecter shutdown clearing up homeless session... 2024-03-29T11:21:22.224 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:22.248+0000 7f8053cf6640 10 client.4248.objecter successfully canceled tick 2024-03-29T11:21:22.224 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:22.248+0000 7f8053cf6640 1 -- 172.21.15.114:0/3657765178 >> v1:172.21.15.114:6816/4038755055 conn(0x7f803006dd60 legacy=0x7f8030070220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:22.224 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:22.248+0000 7f8053cf6640 10 monclient: shutdown 2024-03-29T11:21:22.225 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:22.248+0000 7f8053cf6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:22.225 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:22.248+0000 7f8053cf6640 1 -- 172.21.15.114:0/3657765178 >> v1:172.21.15.114:6789/0 conn(0x7f804c142690 legacy=0x7f804c1e1d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:22.225 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:22.248+0000 7f8053cf6640 1 -- 172.21.15.114:0/3657765178 shutdown_connections 2024-03-29T11:21:22.225 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:22.248+0000 7f8053cf6640 1 -- 172.21.15.114:0/3657765178 >> 172.21.15.114:0/3657765178 conn(0x7f804c066a00 msgr2=0x7f804c0b2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:22.226 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:22.249+0000 7f8053cf6640 1 -- 172.21.15.114:0/3657765178 shutdown_connections 2024-03-29T11:21:22.226 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:22.250+0000 7f8053cf6640 1 -- 172.21.15.114:0/3657765178 wait complete. 2024-03-29T11:21:22.226 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:22.250+0000 7f8053cf6640 1 librados: shutdown 2024-03-29T11:21:22.243 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:29.051 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:29.202 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.222+0000 7f1083ae2640 10 monclient: get_monmap_and_config 2024-03-29T11:21:29.202 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.222+0000 7f1083ae2640 10 monclient: build_initial_monmap 2024-03-29T11:21:29.203 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.222+0000 7f1083ae2640 10 monclient: monmap: 2024-03-29T11:21:29.203 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:21:29.203 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:21:29.203 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:21:29.223971+0000 2024-03-29T11:21:29.203 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:21:29.223971+0000 2024-03-29T11:21:29.203 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:29.203 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:21:29.203 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:21:29.203 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T11:21:29.203 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T11:21:29.203 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:21:29.203 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.224+0000 7f1083ae2640 1 Processor -- start 2024-03-29T11:21:29.204 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.224+0000 7f1083ae2640 1 -- start start 2024-03-29T11:21:29.204 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.224+0000 7f1083ae2640 10 monclient: init 2024-03-29T11:21:29.204 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.225+0000 7f1083ae2640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:29.207 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.225+0000 7f1083ae2640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:21:29.207 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.225+0000 7f1083ae2640 10 monclient(hunting): picked mon.noname-c con 0x7f107c0b4330 addr v1:172.21.15.176:6789/0 2024-03-29T11:21:29.207 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.225+0000 7f1083ae2640 10 monclient(hunting): picked mon.noname-a con 0x7f107c1470b0 addr v1:172.21.15.114:6789/0 2024-03-29T11:21:29.207 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.225+0000 7f1083ae2640 10 monclient(hunting): picked mon.noname-b con 0x7f107c1439b0 addr v1:172.21.15.114:6790/0 2024-03-29T11:21:29.207 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.225+0000 7f1083ae2640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f107c06f2a0 con 0x7f107c1470b0 2024-03-29T11:21:29.208 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.225+0000 7f1083ae2640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f107c146b90 con 0x7f107c0b4330 2024-03-29T11:21:29.208 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.225+0000 7f1083ae2640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f107c0b2ca0 con 0x7f107c1439b0 2024-03-29T11:21:29.208 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.225+0000 7f1083ae2640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:29.208 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.225+0000 7f1083ae2640 10 monclient(hunting): authenticate will time out at 959.407621s 2024-03-29T11:21:29.208 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.227+0000 7f108207f640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f107c1439b0 0x7f107c151d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:50839/0 (socket says 172.21.15.114:50839) 2024-03-29T11:21:29.208 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.227+0000 7f108207f640 1 -- 172.21.15.114:0/4212611922 learned_addr learned my addr 172.21.15.114:0/4212611922 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:21:29.208 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.230+0000 7f108087c640 1 -- 172.21.15.114:0/4212611922 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3844307462 0 0) 0x7f107c0b2ca0 con 0x7f107c1439b0 2024-03-29T11:21:29.208 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.230+0000 7f108087c640 10 monclient(hunting): _negotiate 2024-03-29T11:21:29.209 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.230+0000 7f108087c640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:29.209 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.230+0000 7f108087c640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:29.209 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.230+0000 7f108087c640 10 monclient(hunting): my global_id is 4247 2024-03-29T11:21:29.209 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.230+0000 7f108087c640 1 -- 172.21.15.114:0/4212611922 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1068003520 con 0x7f107c1439b0 2024-03-29T11:21:29.209 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.230+0000 7f108087c640 1 -- 172.21.15.114:0/4212611922 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3945086384 0 0) 0x7f1068003520 con 0x7f107c1439b0 2024-03-29T11:21:29.209 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.231+0000 7f108087c640 1 -- 172.21.15.114:0/4212611922 >> v1:172.21.15.176:6789/0 conn(0x7f107c0b4330 legacy=0x7f107c0b4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:29.209 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.231+0000 7f108087c640 1 -- 172.21.15.114:0/4212611922 >> v1:172.21.15.114:6789/0 conn(0x7f107c1470b0 legacy=0x7f107c0b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:21:29.209 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.231+0000 7f108087c640 10 monclient: _finish_hunting 0 2024-03-29T11:21:29.209 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.231+0000 7f108087c640 1 monclient: found mon.noname-b 2024-03-29T11:21:29.209 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.231+0000 7f108087c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:29.209 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.231+0000 7f108087c640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.114:6790/0 2024-03-29T11:21:29.209 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.231+0000 7f108087c640 1 -- 172.21.15.114:0/4212611922 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f107c06e0c0 con 0x7f107c1439b0 2024-03-29T11:21:29.209 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.231+0000 7f108087c640 10 monclient: _finish_auth 0 2024-03-29T11:21:29.209 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.231+0000 7f108087c640 10 monclient: _check_auth_tickets 2024-03-29T11:21:29.209 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.231+0000 7f108087c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:29.210 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.231+0000 7f1083ae2640 5 monclient: authenticate success, global_id 4247 2024-03-29T11:21:29.210 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.231+0000 7f1083ae2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:29.210 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.231+0000 7f108087c640 1 -- 172.21.15.114:0/4212611922 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f1074001a10 con 0x7f107c1439b0 2024-03-29T11:21:29.210 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.231+0000 7f108087c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:29.210 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.231+0000 7f108087c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:21:29.210 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.231+0000 7f108087c640 10 monclient: dump: 2024-03-29T11:21:29.210 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:21:29.210 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:21:29.210 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:21:29.210 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:21:29.210 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:29.211 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:21:29.211 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:21:29.211 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:21:29.211 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:21:29.211 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:21:29.211 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.231+0000 7f1083ae2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:29.211 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.231+0000 7f108087c640 1 -- 172.21.15.114:0/4212611922 <== mon.2 v1:172.21.15.114:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1074001d20 con 0x7f107c1439b0 2024-03-29T11:21:29.211 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.231+0000 7f108087c640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:29.211 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.231+0000 7f1083ae2640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:29.211 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.231+0000 7f1083ae2640 4 set_mon_vals no callback set 2024-03-29T11:21:29.211 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.232+0000 7f108087c640 1 -- 172.21.15.114:0/4212611922 <== mon.2 v1:172.21.15.114:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f10740020b0 con 0x7f107c1439b0 2024-03-29T11:21:29.211 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.232+0000 7f108087c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:29.211 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.232+0000 7f108087c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:21:29.211 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.232+0000 7f108087c640 10 monclient: dump: 2024-03-29T11:21:29.211 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:21:29.211 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:21:29.212 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:21:29.212 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:21:29.212 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:29.212 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:21:29.212 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:21:29.212 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:21:29.212 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:21:29.212 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:21:29.214 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.232+0000 7f1083ae2640 10 monclient: shutdown 2024-03-29T11:21:29.214 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.232+0000 7f1083ae2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:29.214 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.232+0000 7f1083ae2640 1 -- 172.21.15.114:0/4212611922 >> v1:172.21.15.114:6790/0 conn(0x7f107c1439b0 legacy=0x7f107c151d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:29.214 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.232+0000 7f1083ae2640 1 -- 172.21.15.114:0/4212611922 shutdown_connections 2024-03-29T11:21:29.214 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.232+0000 7f1083ae2640 1 -- 172.21.15.114:0/4212611922 >> 172.21.15.114:0/4212611922 conn(0x7f107c066a00 msgr2=0x7f107c066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:29.214 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.233+0000 7f1083ae2640 1 -- 172.21.15.114:0/4212611922 shutdown_connections 2024-03-29T11:21:29.214 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.233+0000 7f1083ae2640 1 -- 172.21.15.114:0/4212611922 wait complete. 2024-03-29T11:21:29.214 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.234+0000 7f1083ae2640 10 monclient: build_initial_monmap 2024-03-29T11:21:29.214 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.234+0000 7f1083ae2640 10 monclient: monmap: 2024-03-29T11:21:29.214 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:21:29.214 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:29.215 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T11:21:29.215 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T11:21:29.215 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:29.215 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:21:29.215 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:21:29.215 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T11:21:29.215 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T11:21:29.215 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:21:29.215 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.234+0000 7f1083ae2640 1 librados: starting msgr at 2024-03-29T11:21:29.215 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.234+0000 7f1083ae2640 1 librados: starting objecter 2024-03-29T11:21:29.215 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.234+0000 7f1083ae2640 1 Processor -- start 2024-03-29T11:21:29.215 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.235+0000 7f1083ae2640 1 -- start start 2024-03-29T11:21:29.215 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.235+0000 7f1083ae2640 1 librados: setting wanted keys 2024-03-29T11:21:29.215 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.235+0000 7f1083ae2640 1 librados: calling monclient init 2024-03-29T11:21:29.216 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.235+0000 7f1083ae2640 10 monclient: init 2024-03-29T11:21:29.216 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.236+0000 7f1083ae2640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:29.216 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.236+0000 7f1083ae2640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:21:29.216 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.236+0000 7f1083ae2640 10 monclient(hunting): picked mon.noname-c con 0x7f107c0b4330 addr v1:172.21.15.114:6790/0 2024-03-29T11:21:29.216 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.236+0000 7f1083ae2640 10 monclient(hunting): picked mon.noname-a con 0x7f107c1439b0 addr v1:172.21.15.114:6789/0 2024-03-29T11:21:29.218 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.241+0000 7f1083ae2640 10 monclient(hunting): picked mon.noname-b con 0x7f107c1470b0 addr v1:172.21.15.176:6789/0 2024-03-29T11:21:29.218 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.241+0000 7f1083ae2640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f107c1deac0 con 0x7f107c1439b0 2024-03-29T11:21:29.218 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.241+0000 7f1083ae2640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f107c127830 con 0x7f107c1470b0 2024-03-29T11:21:29.218 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.241+0000 7f1083ae2640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f107c122700 con 0x7f107c0b4330 2024-03-29T11:21:29.218 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.241+0000 7f1083ae2640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:29.218 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.241+0000 7f1083ae2640 10 monclient(hunting): authenticate will time out at 959.423512s 2024-03-29T11:21:29.219 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.242+0000 7f108187e640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f107c0b4330 0x7f107c1253d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:50925/0 (socket says 172.21.15.114:50925) 2024-03-29T11:21:29.219 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.242+0000 7f108187e640 1 -- 172.21.15.114:0/2874690802 learned_addr learned my addr 172.21.15.114:0/2874690802 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:21:29.219 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.242+0000 7f10727fc640 10 client.?.objecter ms_handle_connect 0x7f107c0b4330 2024-03-29T11:21:29.219 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.242+0000 7f10727fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:29.219 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.242+0000 7f10727fc640 1 -- 172.21.15.114:0/2874690802 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1521271069 0 0) 0x7f107c122700 con 0x7f107c0b4330 2024-03-29T11:21:29.219 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.242+0000 7f10727fc640 10 monclient(hunting): _negotiate 2024-03-29T11:21:29.219 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.242+0000 7f10727fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:29.219 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.242+0000 7f10727fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:29.219 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.242+0000 7f10727fc640 10 monclient(hunting): my global_id is 4250 2024-03-29T11:21:29.219 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.242+0000 7f10727fc640 1 -- 172.21.15.114:0/2874690802 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1060004eb0 con 0x7f107c0b4330 2024-03-29T11:21:29.220 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.243+0000 7f10727fc640 10 client.?.objecter ms_handle_connect 0x7f107c1470b0 2024-03-29T11:21:29.220 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.243+0000 7f10727fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:29.220 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.243+0000 7f10727fc640 1 -- 172.21.15.114:0/2874690802 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3361065904 0 0) 0x7f107c127830 con 0x7f107c1470b0 2024-03-29T11:21:29.220 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.243+0000 7f10727fc640 10 monclient(hunting): _negotiate 2024-03-29T11:21:29.220 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.243+0000 7f10727fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:29.220 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.243+0000 7f10727fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:29.220 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.243+0000 7f10727fc640 10 monclient(hunting): my global_id is 4249 2024-03-29T11:21:29.221 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.243+0000 7f10727fc640 1 -- 172.21.15.114:0/2874690802 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f107c122700 con 0x7f107c1470b0 2024-03-29T11:21:29.221 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.243+0000 7f10727fc640 1 -- 172.21.15.114:0/2874690802 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1506445612 0 0) 0x7f1060004eb0 con 0x7f107c0b4330 2024-03-29T11:21:29.221 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.243+0000 7f10727fc640 1 -- 172.21.15.114:0/2874690802 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f107c127830 con 0x7f107c0b4330 2024-03-29T11:21:29.221 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.243+0000 7f10727fc640 1 -- 172.21.15.114:0/2874690802 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f106c0018b0 con 0x7f107c0b4330 2024-03-29T11:21:29.221 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.243+0000 7f10727fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:29.221 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.243+0000 7f10727fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:21:29.221 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.243+0000 7f10727fc640 10 monclient(hunting): dump: 2024-03-29T11:21:29.221 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:21:29.221 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:21:29.221 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:21:29.221 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:21:29.221 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:29.221 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:21:29.221 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:21:29.222 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:21:29.222 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:21:29.222 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:21:29.222 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.243+0000 7f10727fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:29.222 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.244+0000 7f10727fc640 1 -- 172.21.15.114:0/2874690802 <== mon.2 v1:172.21.15.114:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1156099100 0 0) 0x7f107c127830 con 0x7f107c0b4330 2024-03-29T11:21:29.222 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.244+0000 7f10727fc640 1 -- 172.21.15.114:0/2874690802 >> v1:172.21.15.176:6789/0 conn(0x7f107c1470b0 legacy=0x7f107c1e5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:29.224 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.244+0000 7f10727fc640 1 -- 172.21.15.114:0/2874690802 >> v1:172.21.15.114:6789/0 conn(0x7f107c1439b0 legacy=0x7f107c1def50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:29.224 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.244+0000 7f10727fc640 10 monclient: _finish_hunting 0 2024-03-29T11:21:29.224 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.244+0000 7f10727fc640 1 monclient: found mon.c 2024-03-29T11:21:29.224 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.244+0000 7f10727fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:29.224 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.244+0000 7f10727fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:21:29.224 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.244+0000 7f10727fc640 1 -- 172.21.15.114:0/2874690802 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f107c122890 con 0x7f107c0b4330 2024-03-29T11:21:29.224 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.244+0000 7f10727fc640 10 monclient: _finish_auth 0 2024-03-29T11:21:29.225 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.244+0000 7f10727fc640 10 monclient: _check_auth_tickets 2024-03-29T11:21:29.225 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.244+0000 7f10727fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:29.225 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.244+0000 7f1083ae2640 5 monclient: authenticate success, global_id 4250 2024-03-29T11:21:29.225 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.244+0000 7f1083ae2640 10 monclient: _renew_subs 2024-03-29T11:21:29.225 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.244+0000 7f1083ae2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:21:29.225 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.244+0000 7f1083ae2640 1 -- 172.21.15.114:0/2874690802 --> v1:172.21.15.114:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f107c1df660 con 0x7f107c0b4330 2024-03-29T11:21:29.225 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.244+0000 7f1083ae2640 10 client.4250.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:29.225 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.244+0000 7f1083ae2640 10 monclient: _renew_subs 2024-03-29T11:21:29.225 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.244+0000 7f1083ae2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:21:29.225 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.244+0000 7f1083ae2640 1 -- 172.21.15.114:0/2874690802 --> v1:172.21.15.114:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f107c126330 con 0x7f107c0b4330 2024-03-29T11:21:29.225 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.244+0000 7f1083ae2640 1 librados: init done 2024-03-29T11:21:29.225 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.245+0000 7f10727fc640 1 -- 172.21.15.114:0/2874690802 <== mon.2 v1:172.21.15.114:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f106c001a10 con 0x7f107c0b4330 2024-03-29T11:21:29.225 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.245+0000 7f10727fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:29.225 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.245+0000 7f10727fc640 1 -- 172.21.15.114:0/2874690802 <== mon.2 v1:172.21.15.114:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f106c0037b0 con 0x7f107c0b4330 2024-03-29T11:21:29.226 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.245+0000 7f10727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:29.226 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.245+0000 7f10727fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:21:29.226 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.245+0000 7f10727fc640 10 monclient: dump: 2024-03-29T11:21:29.226 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:21:29.226 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:21:29.226 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:21:29.226 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:21:29.226 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:29.226 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:21:29.226 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:21:29.226 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:21:29.226 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:21:29.226 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:21:29.226 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.245+0000 7f1073fff640 4 set_mon_vals no callback set 2024-03-29T11:21:29.226 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.247+0000 7f10727fc640 1 -- 172.21.15.114:0/2874690802 <== mon.2 v1:172.21.15.114:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1594292461 0 0) 0x7f106c003990 con 0x7f107c0b4330 2024-03-29T11:21:29.227 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.248+0000 7f10727fc640 1 -- 172.21.15.114:0/2874690802 <== mon.2 v1:172.21.15.114:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3238987313 0 0) 0x7f106c084950 con 0x7f107c0b4330 2024-03-29T11:21:29.227 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.248+0000 7f10727fc640 10 client.4250.objecter ms_dispatch 0x7f107c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:29.227 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.248+0000 7f10727fc640 3 client.4250.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:29.227 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.248+0000 7f10727fc640 3 client.4250.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:29.227 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.248+0000 7f1083ae2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:29.227 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.248+0000 7f1083ae2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:29.227 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.248+0000 7f1083ae2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:21:29.227 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.248+0000 7f1083ae2640 1 -- 172.21.15.114:0/2874690802 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1064005080 con 0x7f107c0b4330 2024-03-29T11:21:29.227 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.248+0000 7f108107d640 10 monclient: get_auth_request con 0x7f106006dd00 auth_method 0 2024-03-29T11:21:29.227 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.248+0000 7f10727fc640 20 client.4250.objecter dump_active .. 0 homeless 2024-03-29T11:21:29.227 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.249+0000 7f10727fc640 10 client.4250.objecter ms_handle_connect 0x7f106006dd00 2024-03-29T11:21:29.238 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.261+0000 7f10727fc640 1 -- 172.21.15.114:0/2874690802 <== mon.2 v1:172.21.15.114:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f106c050060 con 0x7f107c0b4330 2024-03-29T11:21:29.238 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.261+0000 7f10727fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:29.238 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.261+0000 7f10727fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:29.573 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.597+0000 7f1083ae2640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:29.574 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.597+0000 7f1083ae2640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:29.574 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.597+0000 7f1083ae2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:21:29.574 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.597+0000 7f1083ae2640 1 -- 172.21.15.114:0/2874690802 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1064005330 con 0x7f107c0b4330 2024-03-29T11:21:29.575 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.598+0000 7f10727fc640 1 -- 172.21.15.114:0/2874690802 <== mon.2 v1:172.21.15.114:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2858224613) 0x7f106c052160 con 0x7f107c0b4330 2024-03-29T11:21:29.575 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.598+0000 7f10727fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:29.575 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.598+0000 7f10727fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:29.575 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T11:21:29.575 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"b0de2b9b-c45a-44c7-b6ee-e83b39a1e108","created":"2024-03-29T11:20:49.420660+0000","modified":"2024-03-29T11:20:51.700167+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:51.700167+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a64785d3-6b11-499a-b706-cc827ab96a74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5882dadb-e7a8-4604-ba9a-418013c106c8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d7e8c409-bc66-4e4a-bbdb-420bad31388e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0682989-3595-4cd6-90a5-ab9111c278e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"67068120-e3ff-4817-bd06-44feb7479a35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09e8b-8a85-4997-bdea-75d754e6ee67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2f0d9ead-f47f-4cb3-92a4-0a5ad23cd831","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fdb6a25f-6bd8-4c65-9b9b-b460ba7673a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:29.579 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.602+0000 7f1083ae2640 10 librados: watch_flush enter 2024-03-29T11:21:29.579 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.602+0000 7f1083ae2640 10 librados: watch_flush exit 2024-03-29T11:21:29.579 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.602+0000 7f1083ae2640 20 client.4250.objecter shutdown clearing up homeless session... 2024-03-29T11:21:29.579 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.602+0000 7f1083ae2640 10 client.4250.objecter successfully canceled tick 2024-03-29T11:21:29.579 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.602+0000 7f1083ae2640 1 -- 172.21.15.114:0/2874690802 >> v1:172.21.15.114:6816/4038755055 conn(0x7f106006dd00 legacy=0x7f10600701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:29.580 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.602+0000 7f1083ae2640 10 monclient: shutdown 2024-03-29T11:21:29.580 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.602+0000 7f1083ae2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:29.580 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.602+0000 7f1083ae2640 1 -- 172.21.15.114:0/2874690802 >> v1:172.21.15.114:6790/0 conn(0x7f107c0b4330 legacy=0x7f107c1253d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:29.580 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.602+0000 7f1083ae2640 1 -- 172.21.15.114:0/2874690802 shutdown_connections 2024-03-29T11:21:29.580 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.602+0000 7f1083ae2640 1 -- 172.21.15.114:0/2874690802 >> 172.21.15.114:0/2874690802 conn(0x7f107c066a00 msgr2=0x7f107c070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:29.580 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.602+0000 7f1083ae2640 1 -- 172.21.15.114:0/2874690802 shutdown_connections 2024-03-29T11:21:29.580 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.602+0000 7f1083ae2640 1 -- 172.21.15.114:0/2874690802 wait complete. 2024-03-29T11:21:29.580 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:29.603+0000 7f1083ae2640 1 librados: shutdown 2024-03-29T11:21:29.596 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:36.403 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:36.547 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.571+0000 7f9aaafc8640 10 monclient: get_monmap_and_config 2024-03-29T11:21:36.548 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.571+0000 7f9aaafc8640 10 monclient: build_initial_monmap 2024-03-29T11:21:36.548 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.571+0000 7f9aaafc8640 10 monclient: monmap: 2024-03-29T11:21:36.548 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:21:36.548 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:21:36.548 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:21:36.572480+0000 2024-03-29T11:21:36.548 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:21:36.572480+0000 2024-03-29T11:21:36.548 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:36.548 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:21:36.548 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:21:36.548 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T11:21:36.548 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T11:21:36.548 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:21:36.549 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.572+0000 7f9aaafc8640 1 Processor -- start 2024-03-29T11:21:36.549 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.572+0000 7f9aaafc8640 1 -- start start 2024-03-29T11:21:36.549 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.572+0000 7f9aaafc8640 10 monclient: init 2024-03-29T11:21:36.550 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.573+0000 7f9aaafc8640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:36.550 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.573+0000 7f9aaafc8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:21:36.555 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.577+0000 7f9aa9fc6640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f9aa40b4820 0x7f9aa413baf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:48263/0 (socket says 172.21.15.114:48263) 2024-03-29T11:21:36.555 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.577+0000 7f9aa9fc6640 1 -- 172.21.15.114:0/2578961881 learned_addr learned my addr 172.21.15.114:0/2578961881 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:21:36.555 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.577+0000 7f9aaafc8640 10 monclient(hunting): picked mon.noname-b con 0x7f9aa40b4820 addr v1:172.21.15.114:6790/0 2024-03-29T11:21:36.555 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.577+0000 7f9aaafc8640 10 monclient(hunting): picked mon.noname-c con 0x7f9aa40a9ce0 addr v1:172.21.15.176:6789/0 2024-03-29T11:21:36.555 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.577+0000 7f9aaafc8640 10 monclient(hunting): picked mon.noname-a con 0x7f9aa40b3cd0 addr v1:172.21.15.114:6789/0 2024-03-29T11:21:36.555 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.577+0000 7f9aaafc8640 1 -- 172.21.15.114:0/2578961881 --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9aa40b4180 con 0x7f9aa40b3cd0 2024-03-29T11:21:36.555 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.577+0000 7f9aaafc8640 1 -- 172.21.15.114:0/2578961881 --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9aa40aaf60 con 0x7f9aa40a9ce0 2024-03-29T11:21:36.555 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.578+0000 7f9aaafc8640 1 -- 172.21.15.114:0/2578961881 --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9aa413c200 con 0x7f9aa40b4820 2024-03-29T11:21:36.555 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.578+0000 7f9aaafc8640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:36.556 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.578+0000 7f9aaafc8640 10 monclient(hunting): authenticate will time out at 966.760340s 2024-03-29T11:21:36.556 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.578+0000 7f9aa8fc4640 1 -- 172.21.15.114:0/2578961881 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2813032609 0 0) 0x7f9aa413c200 con 0x7f9aa40b4820 2024-03-29T11:21:36.556 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.578+0000 7f9aa8fc4640 10 monclient(hunting): _negotiate 2024-03-29T11:21:36.556 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.578+0000 7f9aa8fc4640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:36.556 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.578+0000 7f9aa8fc4640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:36.556 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.578+0000 7f9aa8fc4640 10 monclient(hunting): my global_id is 4253 2024-03-29T11:21:36.556 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.578+0000 7f9aa8fc4640 1 -- 172.21.15.114:0/2578961881 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a90003520 con 0x7f9aa40b4820 2024-03-29T11:21:36.556 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.578+0000 7f9aa8fc4640 1 -- 172.21.15.114:0/2578961881 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3345510439 0 0) 0x7f9a90003520 con 0x7f9aa40b4820 2024-03-29T11:21:36.556 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.578+0000 7f9aa8fc4640 1 -- 172.21.15.114:0/2578961881 >> v1:172.21.15.176:6789/0 conn(0x7f9aa40a9ce0 legacy=0x7f9aa4145770 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:21:36.561 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.585+0000 7f9aa8fc4640 1 -- 172.21.15.114:0/2578961881 >> v1:172.21.15.114:6789/0 conn(0x7f9aa40b3cd0 legacy=0x7f9aa414d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:36.563 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.585+0000 7f9aa8fc4640 10 monclient: _finish_hunting 0 2024-03-29T11:21:36.563 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.585+0000 7f9aa8fc4640 1 monclient: found mon.noname-b 2024-03-29T11:21:36.564 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.585+0000 7f9aa8fc4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:36.564 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.585+0000 7f9aa8fc4640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.114:6790/0 2024-03-29T11:21:36.564 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.585+0000 7f9aa8fc4640 1 -- 172.21.15.114:0/2578961881 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9aa4145e80 con 0x7f9aa40b4820 2024-03-29T11:21:36.564 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.585+0000 7f9aa8fc4640 10 monclient: _finish_auth 0 2024-03-29T11:21:36.564 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.585+0000 7f9aa8fc4640 10 monclient: _check_auth_tickets 2024-03-29T11:21:36.564 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.585+0000 7f9aa8fc4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:36.564 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.585+0000 7f9aa8fc4640 1 -- 172.21.15.114:0/2578961881 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f9a9c001a10 con 0x7f9aa40b4820 2024-03-29T11:21:36.564 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.585+0000 7f9aa8fc4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:36.564 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.585+0000 7f9aa8fc4640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:21:36.564 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.585+0000 7f9aa8fc4640 10 monclient: dump: 2024-03-29T11:21:36.564 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:21:36.564 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:21:36.564 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:21:36.564 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:21:36.565 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:36.565 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:21:36.565 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:21:36.565 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:21:36.565 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:21:36.565 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:21:36.565 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.585+0000 7f9aaafc8640 5 monclient: authenticate success, global_id 4253 2024-03-29T11:21:36.565 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.585+0000 7f9aaafc8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:36.565 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.587+0000 7f9aa8fc4640 1 -- 172.21.15.114:0/2578961881 <== mon.2 v1:172.21.15.114:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9a9c001c70 con 0x7f9aa40b4820 2024-03-29T11:21:36.565 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.587+0000 7f9aa8fc4640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:36.565 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.587+0000 7f9aa8fc4640 1 -- 172.21.15.114:0/2578961881 <== mon.2 v1:172.21.15.114:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f9a9c002080 con 0x7f9aa40b4820 2024-03-29T11:21:36.565 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.587+0000 7f9aa8fc4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:36.565 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.587+0000 7f9aa8fc4640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:21:36.565 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.587+0000 7f9aa8fc4640 10 monclient: dump: 2024-03-29T11:21:36.565 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:21:36.566 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:21:36.566 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:21:36.566 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:21:36.566 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:36.566 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:21:36.566 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:21:36.566 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:21:36.566 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:21:36.566 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:21:36.566 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.587+0000 7f9aaafc8640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:36.566 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.587+0000 7f9aaafc8640 4 set_mon_vals no callback set 2024-03-29T11:21:36.567 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.588+0000 7f9aaafc8640 10 monclient: shutdown 2024-03-29T11:21:36.567 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.588+0000 7f9aaafc8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:36.567 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.588+0000 7f9aaafc8640 1 -- 172.21.15.114:0/2578961881 >> v1:172.21.15.114:6790/0 conn(0x7f9aa40b4820 legacy=0x7f9aa413baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:36.568 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.589+0000 7f9aaafc8640 1 -- 172.21.15.114:0/2578961881 shutdown_connections 2024-03-29T11:21:36.568 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.589+0000 7f9aaafc8640 1 -- 172.21.15.114:0/2578961881 >> 172.21.15.114:0/2578961881 conn(0x7f9aa406d070 msgr2=0x7f9aa4066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:36.568 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.590+0000 7f9aaafc8640 1 -- 172.21.15.114:0/2578961881 shutdown_connections 2024-03-29T11:21:36.568 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.590+0000 7f9aaafc8640 1 -- 172.21.15.114:0/2578961881 wait complete. 2024-03-29T11:21:36.568 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.591+0000 7f9aaafc8640 10 monclient: build_initial_monmap 2024-03-29T11:21:36.568 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.591+0000 7f9aaafc8640 10 monclient: monmap: 2024-03-29T11:21:36.568 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:21:36.568 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:36.568 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T11:21:36.568 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T11:21:36.568 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:36.568 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:21:36.568 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:21:36.568 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T11:21:36.568 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T11:21:36.569 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:21:36.570 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.591+0000 7f9aaafc8640 1 librados: starting msgr at 2024-03-29T11:21:36.570 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.592+0000 7f9aaafc8640 1 librados: starting objecter 2024-03-29T11:21:36.570 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.592+0000 7f9aaafc8640 1 Processor -- start 2024-03-29T11:21:36.570 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.592+0000 7f9aaafc8640 1 -- start start 2024-03-29T11:21:36.570 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.592+0000 7f9aaafc8640 1 librados: setting wanted keys 2024-03-29T11:21:36.570 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.592+0000 7f9aaafc8640 1 librados: calling monclient init 2024-03-29T11:21:36.570 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.592+0000 7f9aaafc8640 10 monclient: init 2024-03-29T11:21:36.570 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.593+0000 7f9aaafc8640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:36.570 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.593+0000 7f9aaafc8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:21:36.571 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.593+0000 7f9aaafc8640 10 monclient(hunting): picked mon.noname-c con 0x7f9aa40a9ce0 addr v1:172.21.15.114:6790/0 2024-03-29T11:21:36.571 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.593+0000 7f9aaafc8640 10 monclient(hunting): picked mon.noname-b con 0x7f9aa40b3cd0 addr v1:172.21.15.176:6789/0 2024-03-29T11:21:36.571 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.593+0000 7f9aaafc8640 10 monclient(hunting): picked mon.noname-a con 0x7f9aa40b4820 addr v1:172.21.15.114:6789/0 2024-03-29T11:21:36.571 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.593+0000 7f9aaafc8640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9aa41452a0 con 0x7f9aa40b4820 2024-03-29T11:21:36.571 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.593+0000 7f9aaafc8640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9aa4133650 con 0x7f9aa40b3cd0 2024-03-29T11:21:36.571 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.593+0000 7f9aaafc8640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9aa413b260 con 0x7f9aa40a9ce0 2024-03-29T11:21:36.571 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.593+0000 7f9aaafc8640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:36.571 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.593+0000 7f9aaafc8640 10 monclient(hunting): authenticate will time out at 966.775818s 2024-03-29T11:21:36.572 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.596+0000 7f9aa9fc6640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f9aa40a9ce0 0x7f9aa4134170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:48381/0 (socket says 172.21.15.114:48381) 2024-03-29T11:21:36.572 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.596+0000 7f9aa9fc6640 1 -- 172.21.15.114:0/3768614717 learned_addr learned my addr 172.21.15.114:0/3768614717 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:21:36.573 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.596+0000 7f9a9a7fc640 10 client.?.objecter ms_handle_connect 0x7f9aa40a9ce0 2024-03-29T11:21:36.573 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.596+0000 7f9a9a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:36.574 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.597+0000 7f9a9a7fc640 1 -- 172.21.15.114:0/3768614717 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1267043729 0 0) 0x7f9aa413b260 con 0x7f9aa40a9ce0 2024-03-29T11:21:36.574 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.597+0000 7f9a9a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:21:36.574 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.597+0000 7f9a9a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:36.574 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.597+0000 7f9a9a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:36.574 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.597+0000 7f9a9a7fc640 10 monclient(hunting): my global_id is 4256 2024-03-29T11:21:36.574 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.597+0000 7f9a9a7fc640 1 -- 172.21.15.114:0/3768614717 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a88004eb0 con 0x7f9aa40a9ce0 2024-03-29T11:21:36.576 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.598+0000 7f9a9a7fc640 1 -- 172.21.15.114:0/3768614717 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1768760128 0 0) 0x7f9a88004eb0 con 0x7f9aa40a9ce0 2024-03-29T11:21:36.576 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.598+0000 7f9a9a7fc640 1 -- 172.21.15.114:0/3768614717 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9aa413b260 con 0x7f9aa40a9ce0 2024-03-29T11:21:36.576 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.598+0000 7f9a9a7fc640 1 -- 172.21.15.114:0/3768614717 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f9a9c0019d0 con 0x7f9aa40a9ce0 2024-03-29T11:21:36.576 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.598+0000 7f9a9a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:36.576 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.598+0000 7f9a9a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:21:36.576 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.598+0000 7f9a9a7fc640 10 monclient(hunting): dump: 2024-03-29T11:21:36.576 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:21:36.576 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:21:36.576 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:21:36.576 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:21:36.576 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:36.576 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:21:36.577 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:21:36.577 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:21:36.577 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:21:36.577 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:21:36.577 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.598+0000 7f9a9a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:36.577 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.598+0000 7f9a9a7fc640 1 -- 172.21.15.114:0/3768614717 <== mon.2 v1:172.21.15.114:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1228414957 0 0) 0x7f9aa413b260 con 0x7f9aa40a9ce0 2024-03-29T11:21:36.577 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.598+0000 7f9a9a7fc640 1 -- 172.21.15.114:0/3768614717 >> v1:172.21.15.176:6789/0 conn(0x7f9aa40b3cd0 legacy=0x7f9aa4132830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:36.577 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.598+0000 7f9a9a7fc640 1 -- 172.21.15.114:0/3768614717 >> v1:172.21.15.114:6789/0 conn(0x7f9aa40b4820 legacy=0x7f9aa4132f40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:21:36.577 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.599+0000 7f9a9a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:21:36.577 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.599+0000 7f9a9a7fc640 1 monclient: found mon.c 2024-03-29T11:21:36.577 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.599+0000 7f9a9a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:36.577 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.599+0000 7f9a9a7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:21:36.577 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.599+0000 7f9a9a7fc640 1 -- 172.21.15.114:0/3768614717 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9aa413b540 con 0x7f9aa40a9ce0 2024-03-29T11:21:36.577 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.599+0000 7f9a9a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:21:36.577 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.599+0000 7f9a9a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:21:36.578 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.599+0000 7f9a9a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:36.578 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.599+0000 7f9a9a7fc640 10 client.?.objecter ms_handle_connect 0x7f9aa40b4820 2024-03-29T11:21:36.578 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.599+0000 7f9a9a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:36.578 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.599+0000 7f9a9a7fc640 1 -- 172.21.15.114:0/3768614717 <== mon.2 v1:172.21.15.114:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9a9c004e80 con 0x7f9aa40a9ce0 2024-03-29T11:21:36.578 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.599+0000 7f9a9a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:36.578 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.599+0000 7f9a9a7fc640 1 -- 172.21.15.114:0/3768614717 <== mon.2 v1:172.21.15.114:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f9a9c005250 con 0x7f9aa40a9ce0 2024-03-29T11:21:36.578 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.599+0000 7f9a9a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:36.578 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.599+0000 7f9a9a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:21:36.579 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.599+0000 7f9a9a7fc640 10 monclient: dump: 2024-03-29T11:21:36.579 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:21:36.579 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:21:36.579 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:21:36.579 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:21:36.579 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:36.579 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:21:36.579 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:21:36.579 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:21:36.579 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:21:36.579 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:21:36.579 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.599+0000 7f9a9bfff640 4 set_mon_vals no callback set 2024-03-29T11:21:36.579 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.599+0000 7f9aaafc8640 5 monclient: authenticate success, global_id 4256 2024-03-29T11:21:36.579 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.599+0000 7f9aaafc8640 10 monclient: _renew_subs 2024-03-29T11:21:36.579 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.599+0000 7f9aaafc8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:21:36.580 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.599+0000 7f9aaafc8640 1 -- 172.21.15.114:0/3768614717 --> v1:172.21.15.114:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9aa413b820 con 0x7f9aa40a9ce0 2024-03-29T11:21:36.580 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.599+0000 7f9aaafc8640 10 client.4256.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:36.580 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.599+0000 7f9aaafc8640 10 monclient: _renew_subs 2024-03-29T11:21:36.580 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.599+0000 7f9aaafc8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:21:36.580 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.599+0000 7f9aaafc8640 1 -- 172.21.15.114:0/3768614717 --> v1:172.21.15.114:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9aa4144810 con 0x7f9aa40a9ce0 2024-03-29T11:21:36.580 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.599+0000 7f9aaafc8640 1 librados: init done 2024-03-29T11:21:36.580 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.601+0000 7f9aaafc8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:36.580 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.603+0000 7f9aaafc8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:36.580 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.603+0000 7f9aaafc8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:21:36.581 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.603+0000 7f9aaafc8640 1 -- 172.21.15.114:0/3768614717 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9aa404f730 con 0x7f9aa40a9ce0 2024-03-29T11:21:36.581 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.604+0000 7f9a9a7fc640 1 -- 172.21.15.114:0/3768614717 <== mon.2 v1:172.21.15.114:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1594292461 0 0) 0x7f9a9c01b220 con 0x7f9aa40a9ce0 2024-03-29T11:21:36.581 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.605+0000 7f9a9a7fc640 1 -- 172.21.15.114:0/3768614717 <== mon.2 v1:172.21.15.114:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3238987313 0 0) 0x7f9a9c0850c0 con 0x7f9aa40a9ce0 2024-03-29T11:21:36.581 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.605+0000 7f9a9a7fc640 10 client.4256.objecter ms_dispatch 0x7f9aa4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:36.581 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.605+0000 7f9a9a7fc640 3 client.4256.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:36.581 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.605+0000 7f9a9a7fc640 3 client.4256.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:36.582 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.605+0000 7f9a9a7fc640 20 client.4256.objecter dump_active .. 0 homeless 2024-03-29T11:21:36.582 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.605+0000 7f9aa97c5640 10 monclient: get_auth_request con 0x7f9a8806dae0 auth_method 0 2024-03-29T11:21:36.582 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.605+0000 7f9a9a7fc640 10 client.4256.objecter ms_handle_connect 0x7f9a8806dae0 2024-03-29T11:21:36.595 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.616+0000 7f9a9a7fc640 1 -- 172.21.15.114:0/3768614717 <== mon.2 v1:172.21.15.114:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9a9c0509a0 con 0x7f9aa40a9ce0 2024-03-29T11:21:36.595 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.616+0000 7f9a9a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:36.595 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.616+0000 7f9a9a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:36.918 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.941+0000 7f9aaafc8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:36.918 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.941+0000 7f9aaafc8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:36.918 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.941+0000 7f9aaafc8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:21:36.919 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.941+0000 7f9aaafc8640 1 -- 172.21.15.114:0/3768614717 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9aa4144b00 con 0x7f9aa40a9ce0 2024-03-29T11:21:36.921 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T11:21:36.921 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"b0de2b9b-c45a-44c7-b6ee-e83b39a1e108","created":"2024-03-29T11:20:49.420660+0000","modified":"2024-03-29T11:20:51.700167+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:51.700167+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a64785d3-6b11-499a-b706-cc827ab96a74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5882dadb-e7a8-4604-ba9a-418013c106c8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d7e8c409-bc66-4e4a-bbdb-420bad31388e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0682989-3595-4cd6-90a5-ab9111c278e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"67068120-e3ff-4817-bd06-44feb7479a35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09e8b-8a85-4997-bdea-75d754e6ee67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2f0d9ead-f47f-4cb3-92a4-0a5ad23cd831","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fdb6a25f-6bd8-4c65-9b9b-b460ba7673a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:36.921 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.943+0000 7f9a9a7fc640 1 -- 172.21.15.114:0/3768614717 <== mon.2 v1:172.21.15.114:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2858224613) 0x7f9a9c0528d0 con 0x7f9aa40a9ce0 2024-03-29T11:21:36.921 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.943+0000 7f9a9a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:36.921 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.943+0000 7f9a9a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:36.923 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.946+0000 7f9aaafc8640 10 librados: watch_flush enter 2024-03-29T11:21:36.923 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.946+0000 7f9aaafc8640 10 librados: watch_flush exit 2024-03-29T11:21:36.923 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.946+0000 7f9aaafc8640 20 client.4256.objecter shutdown clearing up homeless session... 2024-03-29T11:21:36.923 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.946+0000 7f9aaafc8640 10 client.4256.objecter successfully canceled tick 2024-03-29T11:21:36.923 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.947+0000 7f9aaafc8640 1 -- 172.21.15.114:0/3768614717 >> v1:172.21.15.114:6816/4038755055 conn(0x7f9a8806dae0 legacy=0x7f9a8806ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:36.923 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.947+0000 7f9aaafc8640 10 monclient: shutdown 2024-03-29T11:21:36.923 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.947+0000 7f9aaafc8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:36.923 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.947+0000 7f9aaafc8640 1 -- 172.21.15.114:0/3768614717 >> v1:172.21.15.114:6790/0 conn(0x7f9aa40a9ce0 legacy=0x7f9aa4134170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:36.924 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.947+0000 7f9aaafc8640 1 -- 172.21.15.114:0/3768614717 shutdown_connections 2024-03-29T11:21:36.924 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.947+0000 7f9aaafc8640 1 -- 172.21.15.114:0/3768614717 >> 172.21.15.114:0/3768614717 conn(0x7f9aa406d070 msgr2=0x7f9aa40b8220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:36.924 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.947+0000 7f9aaafc8640 1 -- 172.21.15.114:0/3768614717 shutdown_connections 2024-03-29T11:21:36.924 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.948+0000 7f9aaafc8640 1 -- 172.21.15.114:0/3768614717 wait complete. 2024-03-29T11:21:36.924 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:36.948+0000 7f9aaafc8640 1 librados: shutdown 2024-03-29T11:21:36.944 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:43.751 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:43.901 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.924+0000 7fc31c70e640 10 monclient: get_monmap_and_config 2024-03-29T11:21:43.901 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.924+0000 7fc31c70e640 10 monclient: build_initial_monmap 2024-03-29T11:21:43.901 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.924+0000 7fc31c70e640 10 monclient: monmap: 2024-03-29T11:21:43.901 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:21:43.901 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:21:43.901 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:21:43.925578+0000 2024-03-29T11:21:43.901 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:21:43.925578+0000 2024-03-29T11:21:43.901 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:43.901 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:21:43.901 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:21:43.901 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T11:21:43.901 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T11:21:43.901 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:21:43.902 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.925+0000 7fc31c70e640 1 Processor -- start 2024-03-29T11:21:43.902 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.925+0000 7fc31c70e640 1 -- start start 2024-03-29T11:21:43.902 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.925+0000 7fc31c70e640 10 monclient: init 2024-03-29T11:21:43.908 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.926+0000 7fc31c70e640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:43.908 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.926+0000 7fc31c70e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:21:43.908 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.926+0000 7fc31c70e640 10 monclient(hunting): picked mon.noname-c con 0x7fc3141457f0 addr v1:172.21.15.176:6789/0 2024-03-29T11:21:43.908 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.926+0000 7fc31c70e640 10 monclient(hunting): picked mon.noname-b con 0x7fc31413f650 addr v1:172.21.15.114:6790/0 2024-03-29T11:21:43.908 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.926+0000 7fc31c70e640 10 monclient(hunting): picked mon.noname-a con 0x7fc3140a9d70 addr v1:172.21.15.114:6789/0 2024-03-29T11:21:43.908 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.926+0000 7fc31c70e640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3140aaef0 con 0x7fc3140a9d70 2024-03-29T11:21:43.908 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.926+0000 7fc31c70e640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc314145bd0 con 0x7fc3141457f0 2024-03-29T11:21:43.908 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.926+0000 7fc31c70e640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc314145eb0 con 0x7fc31413f650 2024-03-29T11:21:43.908 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.926+0000 7fc31c70e640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:43.908 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.926+0000 7fc31c70e640 10 monclient(hunting): authenticate will time out at 974.108580s 2024-03-29T11:21:43.909 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.929+0000 7fc31a4aa640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7fc3141457f0 0x7fc31413ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.114:55243/0 (socket says 172.21.15.114:55243) 2024-03-29T11:21:43.909 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.929+0000 7fc31a4aa640 1 -- 172.21.15.114:0/2592516091 learned_addr learned my addr 172.21.15.114:0/2592516091 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:21:43.909 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.930+0000 7fc3194a8640 1 -- 172.21.15.114:0/2592516091 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3574197443 0 0) 0x7fc314145bd0 con 0x7fc3141457f0 2024-03-29T11:21:43.909 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.930+0000 7fc3194a8640 10 monclient(hunting): _negotiate 2024-03-29T11:21:43.909 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.930+0000 7fc3194a8640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:43.909 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.930+0000 7fc3194a8640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:43.909 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.930+0000 7fc3194a8640 10 monclient(hunting): my global_id is 4252 2024-03-29T11:21:43.909 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.930+0000 7fc3194a8640 1 -- 172.21.15.114:0/2592516091 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc300003520 con 0x7fc3141457f0 2024-03-29T11:21:43.909 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.931+0000 7fc3194a8640 1 -- 172.21.15.114:0/2592516091 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2263242715 0 0) 0x7fc300003520 con 0x7fc3141457f0 2024-03-29T11:21:43.909 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.931+0000 7fc3194a8640 1 -- 172.21.15.114:0/2592516091 >> v1:172.21.15.114:6790/0 conn(0x7fc31413f650 legacy=0x7fc3140b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:21:43.911 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.932+0000 7fc3194a8640 1 -- 172.21.15.114:0/2592516091 >> v1:172.21.15.114:6789/0 conn(0x7fc3140a9d70 legacy=0x7fc31414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:43.911 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.932+0000 7fc3194a8640 10 monclient: _finish_hunting 0 2024-03-29T11:21:43.912 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.932+0000 7fc3194a8640 1 monclient: found mon.noname-c 2024-03-29T11:21:43.912 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.932+0000 7fc3194a8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:43.912 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.932+0000 7fc3194a8640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.176:6789/0 2024-03-29T11:21:43.912 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.932+0000 7fc3194a8640 1 -- 172.21.15.114:0/2592516091 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc314146340 con 0x7fc3141457f0 2024-03-29T11:21:43.912 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.932+0000 7fc3194a8640 10 monclient: _finish_auth 0 2024-03-29T11:21:43.912 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.932+0000 7fc3194a8640 10 monclient: _check_auth_tickets 2024-03-29T11:21:43.912 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.932+0000 7fc3194a8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:43.912 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.932+0000 7fc3194a8640 1 -- 172.21.15.114:0/2592516091 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fc304001690 con 0x7fc3141457f0 2024-03-29T11:21:43.912 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.932+0000 7fc3194a8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:43.912 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.932+0000 7fc3194a8640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:21:43.912 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.932+0000 7fc3194a8640 10 monclient: dump: 2024-03-29T11:21:43.912 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:21:43.912 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:21:43.913 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:21:43.913 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:21:43.913 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:43.913 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:21:43.913 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:21:43.913 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:21:43.913 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:21:43.913 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:21:43.913 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.934+0000 7fc3194a8640 1 -- 172.21.15.114:0/2592516091 <== mon.1 v1:172.21.15.176:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc304001860 con 0x7fc3141457f0 2024-03-29T11:21:43.913 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.934+0000 7fc3194a8640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:43.913 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.934+0000 7fc3194a8640 1 -- 172.21.15.114:0/2592516091 <== mon.1 v1:172.21.15.176:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fc304001c70 con 0x7fc3141457f0 2024-03-29T11:21:43.913 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.934+0000 7fc3194a8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:43.913 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.934+0000 7fc3194a8640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:21:43.913 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.934+0000 7fc3194a8640 10 monclient: dump: 2024-03-29T11:21:43.913 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:21:43.913 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:21:43.914 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:21:43.914 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:21:43.914 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:43.914 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:21:43.914 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:21:43.914 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:21:43.914 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:21:43.914 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:21:43.915 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.935+0000 7fc31c70e640 5 monclient: authenticate success, global_id 4252 2024-03-29T11:21:43.915 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.935+0000 7fc31c70e640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:43.915 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.935+0000 7fc31c70e640 4 set_mon_vals no callback set 2024-03-29T11:21:43.915 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.936+0000 7fc31c70e640 10 monclient: shutdown 2024-03-29T11:21:43.915 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.936+0000 7fc31c70e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:43.915 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.936+0000 7fc31c70e640 1 -- 172.21.15.114:0/2592516091 >> v1:172.21.15.176:6789/0 conn(0x7fc3141457f0 legacy=0x7fc31413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:43.915 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.936+0000 7fc31c70e640 1 -- 172.21.15.114:0/2592516091 shutdown_connections 2024-03-29T11:21:43.915 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.936+0000 7fc31c70e640 1 -- 172.21.15.114:0/2592516091 >> 172.21.15.114:0/2592516091 conn(0x7fc314066a00 msgr2=0x7fc314066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:43.915 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.937+0000 7fc31c70e640 1 -- 172.21.15.114:0/2592516091 shutdown_connections 2024-03-29T11:21:43.918 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.941+0000 7fc31c70e640 1 -- 172.21.15.114:0/2592516091 wait complete. 2024-03-29T11:21:43.918 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.942+0000 7fc31c70e640 10 monclient: build_initial_monmap 2024-03-29T11:21:43.919 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.942+0000 7fc31c70e640 10 monclient: monmap: 2024-03-29T11:21:43.919 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:21:43.919 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:43.919 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T11:21:43.919 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T11:21:43.919 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:43.920 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:21:43.920 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:21:43.920 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T11:21:43.920 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T11:21:43.920 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:21:43.920 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.942+0000 7fc31c70e640 1 librados: starting msgr at 2024-03-29T11:21:43.920 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.942+0000 7fc31c70e640 1 librados: starting objecter 2024-03-29T11:21:43.921 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.943+0000 7fc31c70e640 1 Processor -- start 2024-03-29T11:21:43.921 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.943+0000 7fc31c70e640 1 -- start start 2024-03-29T11:21:43.921 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.943+0000 7fc31c70e640 1 librados: setting wanted keys 2024-03-29T11:21:43.921 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.943+0000 7fc31c70e640 1 librados: calling monclient init 2024-03-29T11:21:43.921 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.943+0000 7fc31c70e640 10 monclient: init 2024-03-29T11:21:43.921 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.943+0000 7fc31c70e640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:43.922 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.943+0000 7fc31c70e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:21:43.922 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.943+0000 7fc31c70e640 10 monclient(hunting): picked mon.noname-c con 0x7fc3140a9d70 addr v1:172.21.15.114:6790/0 2024-03-29T11:21:43.922 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.943+0000 7fc31c70e640 10 monclient(hunting): picked mon.noname-a con 0x7fc31413f650 addr v1:172.21.15.114:6789/0 2024-03-29T11:21:43.922 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.943+0000 7fc31c70e640 10 monclient(hunting): picked mon.noname-b con 0x7fc3141457f0 addr v1:172.21.15.176:6789/0 2024-03-29T11:21:43.922 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.944+0000 7fc31c70e640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3141419f0 con 0x7fc31413f650 2024-03-29T11:21:43.922 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.944+0000 7fc31c70e640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3141db250 con 0x7fc3141457f0 2024-03-29T11:21:43.922 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.944+0000 7fc31c70e640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc314144f40 con 0x7fc3140a9d70 2024-03-29T11:21:43.922 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.944+0000 7fc31c70e640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:43.922 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.944+0000 7fc31c70e640 10 monclient(hunting): authenticate will time out at 974.126125s 2024-03-29T11:21:43.926 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.949+0000 7fc31acab640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7fc3141457f0 0x7fc3141dab40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.114:55333/0 (socket says 172.21.15.114:55333) 2024-03-29T11:21:43.926 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.949+0000 7fc31acab640 1 -- 172.21.15.114:0/3655513566 learned_addr learned my addr 172.21.15.114:0/3655513566 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:21:43.930 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.952+0000 7fc30affd640 10 client.?.objecter ms_handle_connect 0x7fc3141457f0 2024-03-29T11:21:43.930 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.952+0000 7fc30affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:43.932 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.954+0000 7fc30affd640 10 client.?.objecter ms_handle_connect 0x7fc3140a9d70 2024-03-29T11:21:43.932 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.954+0000 7fc30affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:43.932 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.954+0000 7fc30affd640 1 -- 172.21.15.114:0/3655513566 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2327614497 0 0) 0x7fc3141db250 con 0x7fc3141457f0 2024-03-29T11:21:43.932 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.954+0000 7fc30affd640 10 monclient(hunting): _negotiate 2024-03-29T11:21:43.932 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.954+0000 7fc30affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:43.933 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.954+0000 7fc30affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:43.933 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.954+0000 7fc30affd640 10 monclient(hunting): my global_id is 4255 2024-03-29T11:21:43.933 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.954+0000 7fc30affd640 1 -- 172.21.15.114:0/3655513566 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2f8004eb0 con 0x7fc3141457f0 2024-03-29T11:21:43.933 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.954+0000 7fc30affd640 1 -- 172.21.15.114:0/3655513566 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1474525194 0 0) 0x7fc314144f40 con 0x7fc3140a9d70 2024-03-29T11:21:43.933 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.954+0000 7fc30affd640 10 monclient(hunting): _negotiate 2024-03-29T11:21:43.933 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.954+0000 7fc30affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:43.933 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.954+0000 7fc30affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:43.933 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.954+0000 7fc30affd640 10 monclient(hunting): my global_id is 4259 2024-03-29T11:21:43.933 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.954+0000 7fc30affd640 1 -- 172.21.15.114:0/3655513566 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3141db250 con 0x7fc3140a9d70 2024-03-29T11:21:43.933 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.954+0000 7fc30affd640 1 -- 172.21.15.114:0/3655513566 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1284167701 0 0) 0x7fc3141db250 con 0x7fc3140a9d70 2024-03-29T11:21:43.933 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.954+0000 7fc30affd640 1 -- 172.21.15.114:0/3655513566 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc314144f40 con 0x7fc3140a9d70 2024-03-29T11:21:43.933 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.955+0000 7fc30affd640 10 client.?.objecter ms_handle_connect 0x7fc31413f650 2024-03-29T11:21:43.933 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.955+0000 7fc30affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:43.933 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.955+0000 7fc30affd640 1 -- 172.21.15.114:0/3655513566 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fc304001b70 con 0x7fc3140a9d70 2024-03-29T11:21:43.934 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.955+0000 7fc30affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:43.934 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.955+0000 7fc30affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:21:43.934 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.955+0000 7fc30affd640 10 monclient(hunting): dump: 2024-03-29T11:21:43.934 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:21:43.934 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:21:43.934 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:21:43.934 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:21:43.934 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:43.934 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:21:43.934 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:21:43.934 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:21:43.934 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:21:43.934 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:21:43.935 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.955+0000 7fc30affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:43.935 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.955+0000 7fc30affd640 1 -- 172.21.15.114:0/3655513566 <== mon.2 v1:172.21.15.114:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2588151270 0 0) 0x7fc314144f40 con 0x7fc3140a9d70 2024-03-29T11:21:43.935 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.955+0000 7fc30affd640 1 -- 172.21.15.114:0/3655513566 >> v1:172.21.15.176:6789/0 conn(0x7fc3141457f0 legacy=0x7fc3141dab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:43.935 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.955+0000 7fc30affd640 1 -- 172.21.15.114:0/3655513566 >> v1:172.21.15.114:6789/0 conn(0x7fc31413f650 legacy=0x7fc3141da430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:43.935 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.955+0000 7fc30affd640 10 monclient: _finish_hunting 0 2024-03-29T11:21:43.935 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.955+0000 7fc30affd640 1 monclient: found mon.c 2024-03-29T11:21:43.935 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.955+0000 7fc30affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:43.935 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.955+0000 7fc30affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:21:43.935 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.955+0000 7fc30affd640 1 -- 172.21.15.114:0/3655513566 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc314141c40 con 0x7fc3140a9d70 2024-03-29T11:21:43.935 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.955+0000 7fc30affd640 10 monclient: _finish_auth 0 2024-03-29T11:21:43.935 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.955+0000 7fc30affd640 10 monclient: _check_auth_tickets 2024-03-29T11:21:43.935 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.955+0000 7fc30affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:43.935 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.955+0000 7fc30affd640 1 -- 172.21.15.114:0/3655513566 <== mon.2 v1:172.21.15.114:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc304004a90 con 0x7fc3140a9d70 2024-03-29T11:21:43.936 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.955+0000 7fc30affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:43.936 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.955+0000 7fc30bfff640 4 set_mon_vals no callback set 2024-03-29T11:21:43.937 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.956+0000 7fc30affd640 1 -- 172.21.15.114:0/3655513566 <== mon.2 v1:172.21.15.114:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fc304004da0 con 0x7fc3140a9d70 2024-03-29T11:21:43.937 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.956+0000 7fc30affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:43.937 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.956+0000 7fc30affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:21:43.937 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.956+0000 7fc30affd640 10 monclient: dump: 2024-03-29T11:21:43.937 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:21:43.937 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:21:43.937 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:21:43.937 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:21:43.937 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:43.937 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:21:43.937 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:21:43.937 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:21:43.938 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:21:43.938 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:21:43.938 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.957+0000 7fc31c70e640 5 monclient: authenticate success, global_id 4259 2024-03-29T11:21:43.938 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.957+0000 7fc31c70e640 10 monclient: _renew_subs 2024-03-29T11:21:43.938 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.957+0000 7fc31c70e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:21:43.938 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.957+0000 7fc31c70e640 1 -- 172.21.15.114:0/3655513566 --> v1:172.21.15.114:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc314141f20 con 0x7fc3140a9d70 2024-03-29T11:21:43.938 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.957+0000 7fc31c70e640 10 client.4259.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:43.938 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.957+0000 7fc31c70e640 10 monclient: _renew_subs 2024-03-29T11:21:43.938 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.957+0000 7fc31c70e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:21:43.938 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.957+0000 7fc31c70e640 1 -- 172.21.15.114:0/3655513566 --> v1:172.21.15.114:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3141359a0 con 0x7fc3140a9d70 2024-03-29T11:21:43.938 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.957+0000 7fc31c70e640 1 librados: init done 2024-03-29T11:21:43.938 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.959+0000 7fc31c70e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:43.938 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.959+0000 7fc31c70e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:43.938 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.959+0000 7fc31c70e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:21:43.939 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.959+0000 7fc31c70e640 1 -- 172.21.15.114:0/3655513566 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc314075f10 con 0x7fc3140a9d70 2024-03-29T11:21:43.940 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.961+0000 7fc30affd640 1 -- 172.21.15.114:0/3655513566 <== mon.2 v1:172.21.15.114:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1594292461 0 0) 0x7fc30401ad70 con 0x7fc3140a9d70 2024-03-29T11:21:43.940 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.962+0000 7fc30affd640 1 -- 172.21.15.114:0/3655513566 <== mon.2 v1:172.21.15.114:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3238987313 0 0) 0x7fc304084940 con 0x7fc3140a9d70 2024-03-29T11:21:43.940 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.962+0000 7fc30affd640 10 client.4259.objecter ms_dispatch 0x7fc314000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:43.940 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.962+0000 7fc319ca9640 10 monclient: get_auth_request con 0x7fc2f806dc90 auth_method 0 2024-03-29T11:21:43.940 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.962+0000 7fc30affd640 3 client.4259.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:43.940 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.962+0000 7fc30affd640 3 client.4259.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:43.940 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.962+0000 7fc30affd640 20 client.4259.objecter dump_active .. 0 homeless 2024-03-29T11:21:43.940 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.962+0000 7fc30affd640 10 client.4259.objecter ms_handle_connect 0x7fc2f806dc90 2024-03-29T11:21:43.947 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.971+0000 7fc30affd640 1 -- 172.21.15.114:0/3655513566 <== mon.2 v1:172.21.15.114:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc304050220 con 0x7fc3140a9d70 2024-03-29T11:21:43.948 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.971+0000 7fc30affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:43.948 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:43.971+0000 7fc30affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:44.273 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:44.296+0000 7fc31c70e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:44.273 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:44.296+0000 7fc31c70e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:44.273 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:44.296+0000 7fc31c70e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:21:44.273 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:44.296+0000 7fc31c70e640 1 -- 172.21.15.114:0/3655513566 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc314074f50 con 0x7fc3140a9d70 2024-03-29T11:21:44.274 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T11:21:44.274 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"b0de2b9b-c45a-44c7-b6ee-e83b39a1e108","created":"2024-03-29T11:20:49.420660+0000","modified":"2024-03-29T11:20:51.700167+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:51.700167+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a64785d3-6b11-499a-b706-cc827ab96a74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5882dadb-e7a8-4604-ba9a-418013c106c8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d7e8c409-bc66-4e4a-bbdb-420bad31388e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0682989-3595-4cd6-90a5-ab9111c278e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"67068120-e3ff-4817-bd06-44feb7479a35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09e8b-8a85-4997-bdea-75d754e6ee67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2f0d9ead-f47f-4cb3-92a4-0a5ad23cd831","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fdb6a25f-6bd8-4c65-9b9b-b460ba7673a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:44.274 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:44.297+0000 7fc30affd640 1 -- 172.21.15.114:0/3655513566 <== mon.2 v1:172.21.15.114:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2858224613) 0x7fc304052150 con 0x7fc3140a9d70 2024-03-29T11:21:44.274 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:44.297+0000 7fc30affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:44.275 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:44.297+0000 7fc30affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:44.279 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:44.300+0000 7fc31c70e640 10 librados: watch_flush enter 2024-03-29T11:21:44.279 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:44.302+0000 7fc31c70e640 10 librados: watch_flush exit 2024-03-29T11:21:44.279 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:44.302+0000 7fc31c70e640 20 client.4259.objecter shutdown clearing up homeless session... 2024-03-29T11:21:44.279 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:44.302+0000 7fc31c70e640 10 client.4259.objecter successfully canceled tick 2024-03-29T11:21:44.279 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:44.302+0000 7fc31c70e640 1 -- 172.21.15.114:0/3655513566 >> v1:172.21.15.114:6816/4038755055 conn(0x7fc2f806dc90 legacy=0x7fc2f8070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:44.279 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:44.302+0000 7fc31c70e640 10 monclient: shutdown 2024-03-29T11:21:44.279 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:44.302+0000 7fc31c70e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:44.279 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:44.302+0000 7fc31c70e640 1 -- 172.21.15.114:0/3655513566 >> v1:172.21.15.114:6790/0 conn(0x7fc3140a9d70 legacy=0x7fc314142340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:44.279 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:44.303+0000 7fc31c70e640 1 -- 172.21.15.114:0/3655513566 shutdown_connections 2024-03-29T11:21:44.279 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:44.303+0000 7fc31c70e640 1 -- 172.21.15.114:0/3655513566 >> 172.21.15.114:0/3655513566 conn(0x7fc314066a00 msgr2=0x7fc31413b3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:44.280 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:44.303+0000 7fc31c70e640 1 -- 172.21.15.114:0/3655513566 shutdown_connections 2024-03-29T11:21:44.280 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:44.303+0000 7fc31c70e640 1 -- 172.21.15.114:0/3655513566 wait complete. 2024-03-29T11:21:44.280 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:44.303+0000 7fc31c70e640 1 librados: shutdown 2024-03-29T11:21:44.296 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:51.103 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:51.258 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.281+0000 7f4ccf59e640 10 monclient: get_monmap_and_config 2024-03-29T11:21:51.258 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.281+0000 7f4ccf59e640 10 monclient: build_initial_monmap 2024-03-29T11:21:51.258 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.281+0000 7f4ccf59e640 10 monclient: monmap: 2024-03-29T11:21:51.258 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:21:51.258 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:21:51.259 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:21:51.282950+0000 2024-03-29T11:21:51.259 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:21:51.282950+0000 2024-03-29T11:21:51.259 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:51.259 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:21:51.259 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:21:51.259 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T11:21:51.259 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T11:21:51.259 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:21:51.260 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.284+0000 7f4ccf59e640 1 Processor -- start 2024-03-29T11:21:51.260 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.284+0000 7f4ccf59e640 1 -- start start 2024-03-29T11:21:51.260 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.284+0000 7f4ccf59e640 10 monclient: init 2024-03-29T11:21:51.262 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.285+0000 7f4ccf59e640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:51.262 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.285+0000 7f4ccf59e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:21:51.264 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.287+0000 7f4ccf59e640 10 monclient(hunting): picked mon.noname-b con 0x7f4cc8006a70 addr v1:172.21.15.114:6790/0 2024-03-29T11:21:51.264 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.288+0000 7f4ccf59e640 10 monclient(hunting): picked mon.noname-a con 0x7f4cc8033060 addr v1:172.21.15.114:6789/0 2024-03-29T11:21:51.267 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.291+0000 7f4ccf59e640 10 monclient(hunting): picked mon.noname-c con 0x7f4cc80c1e70 addr v1:172.21.15.176:6789/0 2024-03-29T11:21:51.267 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.291+0000 7f4ccf59e640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cc802cfb0 con 0x7f4cc8033060 2024-03-29T11:21:51.268 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.291+0000 7f4ccf59e640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cc802ae70 con 0x7f4cc80c1e70 2024-03-29T11:21:51.268 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.291+0000 7f4ccf59e640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cc80c8110 con 0x7f4cc8006a70 2024-03-29T11:21:51.268 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.291+0000 7f4ccf59e640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:51.268 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.291+0000 7f4ccf59e640 10 monclient(hunting): authenticate will time out at 981.473549s 2024-03-29T11:21:51.268 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.291+0000 7f4ccdd9b640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f4cc8033060 0x7f4cc80c1760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:55269/0 (socket says 172.21.15.114:55269) 2024-03-29T11:21:51.268 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.291+0000 7f4ccdd9b640 1 -- 172.21.15.114:0/3526950078 learned_addr learned my addr 172.21.15.114:0/3526950078 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:21:51.271 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.292+0000 7f4ccd59a640 1 -- 172.21.15.114:0/3526950078 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2783747468 0 0) 0x7f4cc802cfb0 con 0x7f4cc8033060 2024-03-29T11:21:51.271 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.292+0000 7f4ccd59a640 10 monclient(hunting): _negotiate 2024-03-29T11:21:51.271 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.292+0000 7f4ccd59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:51.271 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.292+0000 7f4ccd59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:51.271 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.292+0000 7f4ccd59a640 10 monclient(hunting): my global_id is 4263 2024-03-29T11:21:51.271 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.292+0000 7f4ccd59a640 1 -- 172.21.15.114:0/3526950078 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4cb8005730 con 0x7f4cc8033060 2024-03-29T11:21:51.271 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.292+0000 7f4ccd59a640 1 -- 172.21.15.114:0/3526950078 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2510328429 0 0) 0x7f4cc80c8110 con 0x7f4cc8006a70 2024-03-29T11:21:51.271 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.292+0000 7f4ccd59a640 10 monclient(hunting): _negotiate 2024-03-29T11:21:51.271 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.292+0000 7f4ccd59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:51.271 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.292+0000 7f4ccd59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:51.271 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.292+0000 7f4ccd59a640 10 monclient(hunting): my global_id is 4262 2024-03-29T11:21:51.272 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.292+0000 7f4ccd59a640 1 -- 172.21.15.114:0/3526950078 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4cc802cfb0 con 0x7f4cc8006a70 2024-03-29T11:21:51.272 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.292+0000 7f4ccd59a640 1 -- 172.21.15.114:0/3526950078 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2225481461 0 0) 0x7f4cc802cfb0 con 0x7f4cc8006a70 2024-03-29T11:21:51.272 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.293+0000 7f4ccd59a640 1 -- 172.21.15.114:0/3526950078 >> v1:172.21.15.176:6789/0 conn(0x7f4cc80c1e70 legacy=0x7f4cc80cfda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:21:51.272 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.293+0000 7f4ccd59a640 1 -- 172.21.15.114:0/3526950078 >> v1:172.21.15.114:6789/0 conn(0x7f4cc8033060 legacy=0x7f4cc80c1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:51.272 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.293+0000 7f4ccd59a640 10 monclient: _finish_hunting 0 2024-03-29T11:21:51.272 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.293+0000 7f4ccd59a640 1 monclient: found mon.noname-b 2024-03-29T11:21:51.272 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.293+0000 7f4ccd59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:51.272 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.293+0000 7f4ccd59a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.114:6790/0 2024-03-29T11:21:51.272 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.293+0000 7f4ccd59a640 1 -- 172.21.15.114:0/3526950078 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cc802d200 con 0x7f4cc8006a70 2024-03-29T11:21:51.272 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.293+0000 7f4ccd59a640 10 monclient: _finish_auth 0 2024-03-29T11:21:51.272 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.293+0000 7f4ccd59a640 10 monclient: _check_auth_tickets 2024-03-29T11:21:51.272 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.293+0000 7f4ccd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:51.272 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.293+0000 7f4ccd59a640 1 -- 172.21.15.114:0/3526950078 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f4cc40015a0 con 0x7f4cc8006a70 2024-03-29T11:21:51.272 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.293+0000 7f4ccd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:51.272 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.293+0000 7f4ccd59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:21:51.273 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.293+0000 7f4ccd59a640 10 monclient: dump: 2024-03-29T11:21:51.273 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:21:51.273 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:21:51.273 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:21:51.273 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:21:51.273 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:51.273 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:21:51.273 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:21:51.273 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:21:51.273 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:21:51.273 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:21:51.273 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.293+0000 7f4ccf59e640 5 monclient: authenticate success, global_id 4262 2024-03-29T11:21:51.273 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.294+0000 7f4ccf59e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:51.275 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.295+0000 7f4ccd59a640 1 -- 172.21.15.114:0/3526950078 <== mon.2 v1:172.21.15.114:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4cc40017c0 con 0x7f4cc8006a70 2024-03-29T11:21:51.275 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.295+0000 7f4ccd59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:51.275 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.295+0000 7f4ccd59a640 1 -- 172.21.15.114:0/3526950078 <== mon.2 v1:172.21.15.114:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f4cc4001b50 con 0x7f4cc8006a70 2024-03-29T11:21:51.275 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.296+0000 7f4ccf59e640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:51.275 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.296+0000 7f4ccf59e640 4 set_mon_vals no callback set 2024-03-29T11:21:51.275 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.296+0000 7f4ccd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:51.275 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.296+0000 7f4ccd59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:21:51.275 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.296+0000 7f4ccd59a640 10 monclient: dump: 2024-03-29T11:21:51.276 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:21:51.276 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:21:51.276 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:21:51.276 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:21:51.276 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:51.276 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:21:51.276 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:21:51.276 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:21:51.276 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:21:51.276 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:21:51.276 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.297+0000 7f4ccf59e640 10 monclient: shutdown 2024-03-29T11:21:51.276 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.297+0000 7f4ccf59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:51.276 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.297+0000 7f4ccf59e640 1 -- 172.21.15.114:0/3526950078 >> v1:172.21.15.114:6790/0 conn(0x7f4cc8006a70 legacy=0x7f4cc80c7a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:51.276 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.298+0000 7f4ccf59e640 1 -- 172.21.15.114:0/3526950078 shutdown_connections 2024-03-29T11:21:51.277 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.298+0000 7f4ccf59e640 1 -- 172.21.15.114:0/3526950078 >> 172.21.15.114:0/3526950078 conn(0x7f4cc80058f0 msgr2=0x7f4cc8005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:51.277 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.298+0000 7f4ccf59e640 1 -- 172.21.15.114:0/3526950078 shutdown_connections 2024-03-29T11:21:51.277 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.298+0000 7f4ccf59e640 1 -- 172.21.15.114:0/3526950078 wait complete. 2024-03-29T11:21:51.278 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.300+0000 7f4ccf59e640 10 monclient: build_initial_monmap 2024-03-29T11:21:51.278 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.300+0000 7f4ccf59e640 10 monclient: monmap: 2024-03-29T11:21:51.278 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:21:51.278 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:51.278 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T11:21:51.278 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T11:21:51.278 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:51.278 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:21:51.278 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:21:51.278 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T11:21:51.279 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T11:21:51.279 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:21:51.279 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.300+0000 7f4ccf59e640 1 librados: starting msgr at 2024-03-29T11:21:51.279 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.301+0000 7f4ccf59e640 1 librados: starting objecter 2024-03-29T11:21:51.279 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.301+0000 7f4ccf59e640 1 Processor -- start 2024-03-29T11:21:51.279 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.301+0000 7f4ccf59e640 1 -- start start 2024-03-29T11:21:51.279 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.301+0000 7f4ccf59e640 1 librados: setting wanted keys 2024-03-29T11:21:51.279 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.301+0000 7f4ccf59e640 1 librados: calling monclient init 2024-03-29T11:21:51.279 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.301+0000 7f4ccf59e640 10 monclient: init 2024-03-29T11:21:51.281 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.304+0000 7f4ccf59e640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:51.281 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.304+0000 7f4ccf59e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:21:51.281 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.304+0000 7f4ccf59e640 10 monclient(hunting): picked mon.noname-b con 0x7f4cc8006a70 addr v1:172.21.15.176:6789/0 2024-03-29T11:21:51.281 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.304+0000 7f4ccf59e640 10 monclient(hunting): picked mon.noname-c con 0x7f4cc80c1e70 addr v1:172.21.15.114:6790/0 2024-03-29T11:21:51.281 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.304+0000 7f4ccf59e640 10 monclient(hunting): picked mon.noname-a con 0x7f4cc8031e90 addr v1:172.21.15.114:6789/0 2024-03-29T11:21:51.281 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.304+0000 7f4ccf59e640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cc80c5500 con 0x7f4cc8031e90 2024-03-29T11:21:51.281 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.304+0000 7f4ccf59e640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cc8032930 con 0x7f4cc8006a70 2024-03-29T11:21:51.281 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.304+0000 7f4ccf59e640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cc80c5100 con 0x7f4cc80c1e70 2024-03-29T11:21:51.281 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.304+0000 7f4ccf59e640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:51.282 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.304+0000 7f4ccf59e640 10 monclient(hunting): authenticate will time out at 981.486359s 2024-03-29T11:21:51.285 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.308+0000 7f4cced9d640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f4cc8031e90 0x7f4cc815ba50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:55359/0 (socket says 172.21.15.114:55359) 2024-03-29T11:21:51.285 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.308+0000 7f4cced9d640 1 -- 172.21.15.114:0/1591991970 learned_addr learned my addr 172.21.15.114:0/1591991970 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:21:51.285 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.308+0000 7f4cbeffd640 10 client.?.objecter ms_handle_connect 0x7f4cc8031e90 2024-03-29T11:21:51.285 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.308+0000 7f4cbeffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:51.286 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.309+0000 7f4cbeffd640 1 -- 172.21.15.114:0/1591991970 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3897726361 0 0) 0x7f4cc80c5500 con 0x7f4cc8031e90 2024-03-29T11:21:51.286 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.309+0000 7f4cbeffd640 10 monclient(hunting): _negotiate 2024-03-29T11:21:51.286 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.309+0000 7f4cbeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:51.286 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.309+0000 7f4cbeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:51.286 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.309+0000 7f4cbeffd640 10 monclient(hunting): my global_id is 4266 2024-03-29T11:21:51.286 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.309+0000 7f4cbeffd640 1 -- 172.21.15.114:0/1591991970 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4cb0004eb0 con 0x7f4cc8031e90 2024-03-29T11:21:51.289 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.312+0000 7f4cbeffd640 1 -- 172.21.15.114:0/1591991970 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 240258448 0 0) 0x7f4cb0004eb0 con 0x7f4cc8031e90 2024-03-29T11:21:51.290 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.312+0000 7f4cbeffd640 1 -- 172.21.15.114:0/1591991970 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4cc80c5500 con 0x7f4cc8031e90 2024-03-29T11:21:51.290 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.312+0000 7f4cbeffd640 1 -- 172.21.15.114:0/1591991970 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f4cd004fdd0 con 0x7f4cc8031e90 2024-03-29T11:21:51.290 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.312+0000 7f4cbeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:51.290 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.312+0000 7f4cbeffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:21:51.290 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.312+0000 7f4cbeffd640 10 monclient(hunting): dump: 2024-03-29T11:21:51.290 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:21:51.290 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:21:51.290 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:21:51.290 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:21:51.290 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:51.290 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:21:51.290 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:21:51.290 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:21:51.290 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:21:51.291 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:21:51.291 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.312+0000 7f4cbeffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:51.291 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.312+0000 7f4cbeffd640 1 -- 172.21.15.114:0/1591991970 <== mon.0 v1:172.21.15.114:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2315787696 0 0) 0x7f4cc80c5500 con 0x7f4cc8031e90 2024-03-29T11:21:51.291 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.312+0000 7f4cbeffd640 1 -- 172.21.15.114:0/1591991970 >> v1:172.21.15.114:6790/0 conn(0x7f4cc80c1e70 legacy=0x7f4cc815c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:51.291 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.312+0000 7f4cbeffd640 1 -- 172.21.15.114:0/1591991970 >> v1:172.21.15.176:6789/0 conn(0x7f4cc8006a70 legacy=0x7f4cc802d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:51.291 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.312+0000 7f4cbeffd640 10 monclient: _finish_hunting 0 2024-03-29T11:21:51.291 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.312+0000 7f4cbeffd640 1 monclient: found mon.a 2024-03-29T11:21:51.291 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.313+0000 7f4cbeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:51.291 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.313+0000 7f4cbeffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T11:21:51.291 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.313+0000 7f4cbeffd640 1 -- 172.21.15.114:0/1591991970 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cc8031110 con 0x7f4cc8031e90 2024-03-29T11:21:51.291 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.313+0000 7f4cbeffd640 10 monclient: _finish_auth 0 2024-03-29T11:21:51.291 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.313+0000 7f4cbeffd640 10 monclient: _check_auth_tickets 2024-03-29T11:21:51.291 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.313+0000 7f4cbeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:51.291 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.313+0000 7f4cbeffd640 1 -- 172.21.15.114:0/1591991970 <== mon.0 v1:172.21.15.114:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4cd004f400 con 0x7f4cc8031e90 2024-03-29T11:21:51.291 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.313+0000 7f4cbeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:51.292 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.313+0000 7f4cbeffd640 1 -- 172.21.15.114:0/1591991970 <== mon.0 v1:172.21.15.114:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f4cd0075f10 con 0x7f4cc8031e90 2024-03-29T11:21:51.292 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.313+0000 7f4cbeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:51.292 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.313+0000 7f4cbeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:21:51.292 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.313+0000 7f4cbeffd640 10 monclient: dump: 2024-03-29T11:21:51.292 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:21:51.292 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:21:51.292 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:21:51.292 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:21:51.292 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:51.292 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:21:51.292 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:21:51.292 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:21:51.292 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:21:51.292 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:21:51.292 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.313+0000 7f4cccd99640 4 set_mon_vals no callback set 2024-03-29T11:21:51.294 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.313+0000 7f4ccf59e640 5 monclient: authenticate success, global_id 4266 2024-03-29T11:21:51.294 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.314+0000 7f4ccf59e640 10 monclient: _renew_subs 2024-03-29T11:21:51.294 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.314+0000 7f4ccf59e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T11:21:51.294 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.314+0000 7f4ccf59e640 1 -- 172.21.15.114:0/1591991970 --> v1:172.21.15.114:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cc80313c0 con 0x7f4cc8031e90 2024-03-29T11:21:51.294 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.314+0000 7f4ccf59e640 10 client.4266.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:51.294 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.314+0000 7f4ccf59e640 10 monclient: _renew_subs 2024-03-29T11:21:51.294 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.314+0000 7f4ccf59e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T11:21:51.294 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.314+0000 7f4ccf59e640 1 -- 172.21.15.114:0/1591991970 --> v1:172.21.15.114:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cc8031850 con 0x7f4cc8031e90 2024-03-29T11:21:51.294 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.314+0000 7f4ccf59e640 1 librados: init done 2024-03-29T11:21:51.294 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.316+0000 7f4ccf59e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:51.295 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.316+0000 7f4ccf59e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:51.295 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.316+0000 7f4ccf59e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T11:21:51.295 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.316+0000 7f4ccf59e640 1 -- 172.21.15.114:0/1591991970 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cc802f820 con 0x7f4cc8031e90 2024-03-29T11:21:51.295 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.316+0000 7f4cbeffd640 1 -- 172.21.15.114:0/1591991970 <== mon.0 v1:172.21.15.114:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1594292461 0 0) 0x7f4cd0067bf0 con 0x7f4cc8031e90 2024-03-29T11:21:51.295 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.317+0000 7f4cbeffd640 1 -- 172.21.15.114:0/1591991970 <== mon.0 v1:172.21.15.114:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3238987313 0 0) 0x7f4cd00ff4e0 con 0x7f4cc8031e90 2024-03-29T11:21:51.295 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.317+0000 7f4cbeffd640 10 client.4266.objecter ms_dispatch 0x7f4cd0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:51.295 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.317+0000 7f4cbeffd640 3 client.4266.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:51.295 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.317+0000 7f4cbeffd640 3 client.4266.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:51.295 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.317+0000 7f4cbeffd640 20 client.4266.objecter dump_active .. 0 homeless 2024-03-29T11:21:51.295 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.318+0000 7f4cce59c640 10 monclient: get_auth_request con 0x7f4cb006db90 auth_method 0 2024-03-29T11:21:51.296 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.318+0000 7f4cbeffd640 10 client.4266.objecter ms_handle_connect 0x7f4cb006db90 2024-03-29T11:21:51.307 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.331+0000 7f4cbeffd640 1 -- 172.21.15.114:0/1591991970 <== mon.0 v1:172.21.15.114:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4cd00cae20 con 0x7f4cc8031e90 2024-03-29T11:21:51.307 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.331+0000 7f4cbeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:51.307 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.331+0000 7f4cbeffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:51.631 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.654+0000 7f4ccf59e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:51.631 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.654+0000 7f4ccf59e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:51.631 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.654+0000 7f4ccf59e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T11:21:51.631 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.654+0000 7f4ccf59e640 1 -- 172.21.15.114:0/1591991970 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4cc80c5290 con 0x7f4cc8031e90 2024-03-29T11:21:51.633 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.657+0000 7f4cbeffd640 1 -- 172.21.15.114:0/1591991970 <== mon.0 v1:172.21.15.114:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2858224613) 0x7f4cd00ccd50 con 0x7f4cc8031e90 2024-03-29T11:21:51.633 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.657+0000 7f4cbeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:51.633 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.657+0000 7f4cbeffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:51.635 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T11:21:51.635 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"b0de2b9b-c45a-44c7-b6ee-e83b39a1e108","created":"2024-03-29T11:20:49.420660+0000","modified":"2024-03-29T11:20:51.700167+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:51.700167+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a64785d3-6b11-499a-b706-cc827ab96a74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5882dadb-e7a8-4604-ba9a-418013c106c8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d7e8c409-bc66-4e4a-bbdb-420bad31388e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0682989-3595-4cd6-90a5-ab9111c278e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"67068120-e3ff-4817-bd06-44feb7479a35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09e8b-8a85-4997-bdea-75d754e6ee67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2f0d9ead-f47f-4cb3-92a4-0a5ad23cd831","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fdb6a25f-6bd8-4c65-9b9b-b460ba7673a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:51.638 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.661+0000 7f4ccf59e640 10 librados: watch_flush enter 2024-03-29T11:21:51.638 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.661+0000 7f4ccf59e640 10 librados: watch_flush exit 2024-03-29T11:21:51.638 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.661+0000 7f4ccf59e640 20 client.4266.objecter shutdown clearing up homeless session... 2024-03-29T11:21:51.638 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.661+0000 7f4ccf59e640 10 client.4266.objecter successfully canceled tick 2024-03-29T11:21:51.638 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.661+0000 7f4ccf59e640 1 -- 172.21.15.114:0/1591991970 >> v1:172.21.15.114:6816/4038755055 conn(0x7f4cb006db90 legacy=0x7f4cb0070050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:51.638 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.662+0000 7f4ccf59e640 10 monclient: shutdown 2024-03-29T11:21:51.639 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.662+0000 7f4ccf59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:51.639 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.662+0000 7f4ccf59e640 1 -- 172.21.15.114:0/1591991970 >> v1:172.21.15.114:6789/0 conn(0x7f4cc8031e90 legacy=0x7f4cc815ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:51.640 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.664+0000 7f4ccf59e640 1 -- 172.21.15.114:0/1591991970 shutdown_connections 2024-03-29T11:21:51.641 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.664+0000 7f4ccf59e640 1 -- 172.21.15.114:0/1591991970 >> 172.21.15.114:0/1591991970 conn(0x7f4cc80058f0 msgr2=0x7f4cc80349e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:51.641 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.665+0000 7f4ccf59e640 1 -- 172.21.15.114:0/1591991970 shutdown_connections 2024-03-29T11:21:51.641 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.665+0000 7f4ccf59e640 1 -- 172.21.15.114:0/1591991970 wait complete. 2024-03-29T11:21:51.641 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:51.665+0000 7f4ccf59e640 1 librados: shutdown 2024-03-29T11:21:51.659 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:58.466 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:58.626 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.645+0000 7febb8739640 10 monclient: get_monmap_and_config 2024-03-29T11:21:58.626 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.645+0000 7febb8739640 10 monclient: build_initial_monmap 2024-03-29T11:21:58.626 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.645+0000 7febb8739640 10 monclient: monmap: 2024-03-29T11:21:58.626 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:21:58.626 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:21:58.627 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:21:58.647011+0000 2024-03-29T11:21:58.627 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:21:58.647011+0000 2024-03-29T11:21:58.627 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:58.627 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:21:58.627 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:21:58.627 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T11:21:58.627 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T11:21:58.627 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:21:58.627 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.646+0000 7febb8739640 1 Processor -- start 2024-03-29T11:21:58.627 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.646+0000 7febb8739640 1 -- start start 2024-03-29T11:21:58.627 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.646+0000 7febb8739640 10 monclient: init 2024-03-29T11:21:58.627 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.647+0000 7febb8739640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:58.627 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.647+0000 7febb8739640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:21:58.627 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.647+0000 7febb8739640 10 monclient(hunting): picked mon.noname-c con 0x7febb00ad6c0 addr v1:172.21.15.176:6789/0 2024-03-29T11:21:58.627 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.647+0000 7febb8739640 10 monclient(hunting): picked mon.noname-a con 0x7febb00af6c0 addr v1:172.21.15.114:6789/0 2024-03-29T11:21:58.628 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.647+0000 7febb8739640 10 monclient(hunting): picked mon.noname-b con 0x7febb014fdc0 addr v1:172.21.15.114:6790/0 2024-03-29T11:21:58.628 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.647+0000 7febb8739640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febb00a94e0 con 0x7febb00af6c0 2024-03-29T11:21:58.628 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.647+0000 7febb8739640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febb00af3a0 con 0x7febb00ad6c0 2024-03-29T11:21:58.628 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.647+0000 7febb8739640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febb00afb60 con 0x7febb014fdc0 2024-03-29T11:21:58.628 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.647+0000 7febb8739640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:58.628 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.647+0000 7febb8739640 10 monclient(hunting): authenticate will time out at 988.829980s 2024-03-29T11:21:58.634 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.652+0000 7febb5cd4640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7febb00af6c0 0x7febb0152980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:54329/0 (socket says 172.21.15.114:54329) 2024-03-29T11:21:58.634 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.652+0000 7febb5cd4640 1 -- 172.21.15.114:0/3842876683 learned_addr learned my addr 172.21.15.114:0/3842876683 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:21:58.634 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.655+0000 7febb54d3640 1 -- 172.21.15.114:0/3842876683 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1187007396 0 0) 0x7febb00a94e0 con 0x7febb00af6c0 2024-03-29T11:21:58.634 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.655+0000 7febb54d3640 10 monclient(hunting): _negotiate 2024-03-29T11:21:58.634 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.655+0000 7febb54d3640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:58.634 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.655+0000 7febb54d3640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:58.634 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.655+0000 7febb54d3640 10 monclient(hunting): my global_id is 4269 2024-03-29T11:21:58.634 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.655+0000 7febb54d3640 1 -- 172.21.15.114:0/3842876683 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb9c003520 con 0x7febb00af6c0 2024-03-29T11:21:58.634 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.655+0000 7febb54d3640 1 -- 172.21.15.114:0/3842876683 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3849597448 0 0) 0x7feb9c003520 con 0x7febb00af6c0 2024-03-29T11:21:58.634 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.655+0000 7febb54d3640 1 -- 172.21.15.114:0/3842876683 >> v1:172.21.15.114:6790/0 conn(0x7febb014fdc0 legacy=0x7febb015a8e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:21:58.634 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.656+0000 7febb54d3640 1 -- 172.21.15.114:0/3842876683 >> v1:172.21.15.176:6789/0 conn(0x7febb00ad6c0 legacy=0x7febb014f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:58.634 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.656+0000 7febb54d3640 10 monclient: _finish_hunting 0 2024-03-29T11:21:58.635 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.656+0000 7febb54d3640 1 monclient: found mon.noname-a 2024-03-29T11:21:58.635 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.656+0000 7febb54d3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:58.635 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.656+0000 7febb54d3640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.114:6789/0 2024-03-29T11:21:58.635 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.656+0000 7febb54d3640 1 -- 172.21.15.114:0/3842876683 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febb00aab80 con 0x7febb00af6c0 2024-03-29T11:21:58.635 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.656+0000 7febb54d3640 10 monclient: _finish_auth 0 2024-03-29T11:21:58.635 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.657+0000 7febb54d3640 10 monclient: _check_auth_tickets 2024-03-29T11:21:58.635 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.657+0000 7febb54d3640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:58.635 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.657+0000 7febb54d3640 1 -- 172.21.15.114:0/3842876683 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7feba8001670 con 0x7febb00af6c0 2024-03-29T11:21:58.635 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.657+0000 7febb8739640 5 monclient: authenticate success, global_id 4269 2024-03-29T11:21:58.635 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.657+0000 7febb8739640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:58.635 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.657+0000 7febb54d3640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:58.635 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.657+0000 7febb54d3640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:21:58.635 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.657+0000 7febb54d3640 10 monclient: dump: 2024-03-29T11:21:58.635 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:21:58.635 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:21:58.636 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:21:58.636 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:21:58.636 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:58.636 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:21:58.636 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:21:58.636 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:21:58.636 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:21:58.636 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:21:58.636 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.657+0000 7febb8739640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:58.636 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.657+0000 7febb54d3640 1 -- 172.21.15.114:0/3842876683 <== mon.0 v1:172.21.15.114:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feba8001980 con 0x7febb00af6c0 2024-03-29T11:21:58.636 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.657+0000 7febb54d3640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:58.637 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.657+0000 7febb8739640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:58.637 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.657+0000 7febb8739640 4 set_mon_vals no callback set 2024-03-29T11:21:58.637 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.658+0000 7febb8739640 10 monclient: shutdown 2024-03-29T11:21:58.637 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.658+0000 7febb8739640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:58.637 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.658+0000 7febb8739640 1 -- 172.21.15.114:0/3842876683 >> v1:172.21.15.114:6789/0 conn(0x7febb00af6c0 legacy=0x7febb0152980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:58.637 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.659+0000 7febb8739640 1 -- 172.21.15.114:0/3842876683 shutdown_connections 2024-03-29T11:21:58.638 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.659+0000 7febb8739640 1 -- 172.21.15.114:0/3842876683 >> 172.21.15.114:0/3842876683 conn(0x7febb00b35e0 msgr2=0x7febb00b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:58.638 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.659+0000 7febb8739640 1 -- 172.21.15.114:0/3842876683 shutdown_connections 2024-03-29T11:21:58.638 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.659+0000 7febb8739640 1 -- 172.21.15.114:0/3842876683 wait complete. 2024-03-29T11:21:58.638 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.660+0000 7febb8739640 10 monclient: build_initial_monmap 2024-03-29T11:21:58.638 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.660+0000 7febb8739640 10 monclient: monmap: 2024-03-29T11:21:58.638 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:21:58.638 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:58.638 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T11:21:58.639 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T11:21:58.639 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:58.639 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:21:58.639 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:21:58.639 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T11:21:58.639 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T11:21:58.639 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:21:58.639 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.661+0000 7febb8739640 1 librados: starting msgr at 2024-03-29T11:21:58.639 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.661+0000 7febb8739640 1 librados: starting objecter 2024-03-29T11:21:58.639 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.661+0000 7febb8739640 1 Processor -- start 2024-03-29T11:21:58.644 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.661+0000 7febb8739640 1 -- start start 2024-03-29T11:21:58.644 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.661+0000 7febb8739640 1 librados: setting wanted keys 2024-03-29T11:21:58.644 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.661+0000 7febb8739640 1 librados: calling monclient init 2024-03-29T11:21:58.644 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.661+0000 7febb8739640 10 monclient: init 2024-03-29T11:21:58.644 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.662+0000 7febb8739640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:58.644 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.662+0000 7febb8739640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:21:58.644 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.662+0000 7febb8739640 10 monclient(hunting): picked mon.noname-c con 0x7febb00ad6c0 addr v1:172.21.15.114:6790/0 2024-03-29T11:21:58.644 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.662+0000 7febb8739640 10 monclient(hunting): picked mon.noname-b con 0x7febb00af6c0 addr v1:172.21.15.176:6789/0 2024-03-29T11:21:58.644 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.662+0000 7febb8739640 10 monclient(hunting): picked mon.noname-a con 0x7febb014fdc0 addr v1:172.21.15.114:6789/0 2024-03-29T11:21:58.644 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.662+0000 7febb8739640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febb01f8420 con 0x7febb014fdc0 2024-03-29T11:21:58.644 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.662+0000 7febb8739640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febb007aff0 con 0x7febb00af6c0 2024-03-29T11:21:58.644 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.662+0000 7febb8739640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febb01f92a0 con 0x7febb00ad6c0 2024-03-29T11:21:58.645 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.662+0000 7febb8739640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:58.645 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.662+0000 7febb8739640 10 monclient(hunting): authenticate will time out at 988.844211s 2024-03-29T11:21:58.645 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.666+0000 7febb5cd4640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7febb00af6c0 0x7febb007bcf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.114:52877/0 (socket says 172.21.15.114:52877) 2024-03-29T11:21:58.645 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.666+0000 7febb5cd4640 1 -- 172.21.15.114:0/411402640 learned_addr learned my addr 172.21.15.114:0/411402640 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:21:58.645 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.666+0000 7feba6ffd640 10 client.?.objecter ms_handle_connect 0x7febb00af6c0 2024-03-29T11:21:58.645 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.666+0000 7feba6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:58.645 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.666+0000 7feba6ffd640 1 -- 172.21.15.114:0/411402640 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4128109739 0 0) 0x7febb007aff0 con 0x7febb00af6c0 2024-03-29T11:21:58.645 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.666+0000 7feba6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:21:58.645 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.666+0000 7feba6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:58.645 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.666+0000 7feba6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:58.645 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.667+0000 7feba6ffd640 10 monclient(hunting): my global_id is 4261 2024-03-29T11:21:58.645 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.667+0000 7feba6ffd640 1 -- 172.21.15.114:0/411402640 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb94004eb0 con 0x7febb00af6c0 2024-03-29T11:21:58.645 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.667+0000 7feba6ffd640 10 client.?.objecter ms_handle_connect 0x7febb00ad6c0 2024-03-29T11:21:58.647 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.667+0000 7feba6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:58.647 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.667+0000 7feba6ffd640 1 -- 172.21.15.114:0/411402640 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3250107767 0 0) 0x7febb01f92a0 con 0x7febb00ad6c0 2024-03-29T11:21:58.647 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.667+0000 7feba6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:21:58.647 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.667+0000 7feba6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:58.647 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.667+0000 7feba6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:58.647 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.667+0000 7feba6ffd640 10 monclient(hunting): my global_id is 4265 2024-03-29T11:21:58.647 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.667+0000 7feba6ffd640 1 -- 172.21.15.114:0/411402640 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febb007aff0 con 0x7febb00ad6c0 2024-03-29T11:21:58.647 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.669+0000 7feba6ffd640 1 -- 172.21.15.114:0/411402640 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2289247790 0 0) 0x7feb94004eb0 con 0x7febb00af6c0 2024-03-29T11:21:58.647 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.669+0000 7feba6ffd640 1 -- 172.21.15.114:0/411402640 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7febb01f92a0 con 0x7febb00af6c0 2024-03-29T11:21:58.648 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.669+0000 7feba6ffd640 1 -- 172.21.15.114:0/411402640 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2459830058 0 0) 0x7febb007aff0 con 0x7febb00ad6c0 2024-03-29T11:21:58.648 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.669+0000 7feba6ffd640 1 -- 172.21.15.114:0/411402640 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feb94004eb0 con 0x7febb00ad6c0 2024-03-29T11:21:58.648 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.669+0000 7feba6ffd640 1 -- 172.21.15.114:0/411402640 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7feba8004fb0 con 0x7febb00af6c0 2024-03-29T11:21:58.648 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.669+0000 7feba6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:58.648 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.669+0000 7feba6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:21:58.648 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.669+0000 7feba6ffd640 10 monclient(hunting): dump: 2024-03-29T11:21:58.648 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:21:58.648 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:21:58.649 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:21:58.649 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:21:58.649 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:58.649 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:21:58.649 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:21:58.649 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:21:58.649 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:21:58.649 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:21:58.649 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.669+0000 7feba6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:58.649 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.669+0000 7feba6ffd640 1 -- 172.21.15.114:0/411402640 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7febac001830 con 0x7febb00ad6c0 2024-03-29T11:21:58.649 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.669+0000 7feba6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:58.649 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.669+0000 7feba6ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:21:58.649 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.669+0000 7feba6ffd640 10 monclient(hunting): dump: 2024-03-29T11:21:58.649 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:21:58.650 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:21:58.650 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:21:58.650 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:21:58.650 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:58.650 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:21:58.650 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:21:58.650 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:21:58.650 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:21:58.650 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:21:58.650 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.669+0000 7feba6ffd640 1 -- 172.21.15.114:0/411402640 <== mon.1 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1044024279 0 0) 0x7febb01f92a0 con 0x7febb00af6c0 2024-03-29T11:21:58.651 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.669+0000 7feba6ffd640 1 -- 172.21.15.114:0/411402640 >> v1:172.21.15.114:6790/0 conn(0x7febb00ad6c0 legacy=0x7febb01f8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:58.651 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.669+0000 7feba6ffd640 1 -- 172.21.15.114:0/411402640 >> v1:172.21.15.114:6789/0 conn(0x7febb014fdc0 legacy=0x7febb01ffa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:58.651 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.669+0000 7feba6ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:21:58.651 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.669+0000 7feba6ffd640 1 monclient: found mon.b 2024-03-29T11:21:58.651 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.669+0000 7feba6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:58.651 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.669+0000 7feba6ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:21:58.651 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.669+0000 7feba6ffd640 1 -- 172.21.15.114:0/411402640 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febb01f8670 con 0x7febb00af6c0 2024-03-29T11:21:58.651 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.669+0000 7feba6ffd640 10 monclient: _finish_auth 0 2024-03-29T11:21:58.651 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.669+0000 7feba6ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:21:58.651 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.669+0000 7feba6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:58.651 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.669+0000 7feba6ffd640 1 -- 172.21.15.114:0/411402640 <== mon.1 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feba80053c0 con 0x7febb00af6c0 2024-03-29T11:21:58.651 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.669+0000 7feba6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:58.651 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.669+0000 7febb4cd2640 4 set_mon_vals no callback set 2024-03-29T11:21:58.651 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.670+0000 7febb8739640 5 monclient: authenticate success, global_id 4261 2024-03-29T11:21:58.652 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.670+0000 7febb8739640 10 monclient: _renew_subs 2024-03-29T11:21:58.652 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.670+0000 7febb8739640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:21:58.652 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.670+0000 7febb8739640 1 -- 172.21.15.114:0/411402640 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febb0070970 con 0x7febb00af6c0 2024-03-29T11:21:58.652 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.670+0000 7febb8739640 10 client.4261.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:58.652 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.670+0000 7febb8739640 10 monclient: _renew_subs 2024-03-29T11:21:58.652 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.670+0000 7febb8739640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:21:58.652 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.670+0000 7febb8739640 1 -- 172.21.15.114:0/411402640 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7febb007c400 con 0x7febb00af6c0 2024-03-29T11:21:58.652 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.670+0000 7febb8739640 1 librados: init done 2024-03-29T11:21:58.652 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.670+0000 7feba6ffd640 1 -- 172.21.15.114:0/411402640 <== mon.1 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7feba80056d0 con 0x7febb00af6c0 2024-03-29T11:21:58.652 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.670+0000 7feba6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:58.652 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.670+0000 7feba6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:21:58.652 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.670+0000 7feba6ffd640 10 monclient: dump: 2024-03-29T11:21:58.652 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:21:58.652 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:21:58.652 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:21:58.653 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:21:58.653 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:58.653 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:21:58.653 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:21:58.653 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:21:58.653 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:21:58.653 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:21:58.655 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.674+0000 7febb8739640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:58.655 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.674+0000 7febb8739640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:58.655 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.674+0000 7febb8739640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:21:58.655 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.674+0000 7febb8739640 1 -- 172.21.15.114:0/411402640 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febb01f88b0 con 0x7febb00af6c0 2024-03-29T11:21:58.655 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.675+0000 7feba6ffd640 1 -- 172.21.15.114:0/411402640 <== mon.1 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1594292461 0 0) 0x7feba8001bb0 con 0x7febb00af6c0 2024-03-29T11:21:58.655 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.676+0000 7febb64d5640 10 monclient: get_auth_request con 0x7feb9406de70 auth_method 0 2024-03-29T11:21:58.655 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.677+0000 7feba6ffd640 10 client.4261.objecter ms_handle_connect 0x7feb9406de70 2024-03-29T11:21:58.655 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.677+0000 7feba6ffd640 1 -- 172.21.15.114:0/411402640 <== mon.1 v1:172.21.15.176:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3238987313 0 0) 0x7feba8085190 con 0x7febb00af6c0 2024-03-29T11:21:58.655 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.677+0000 7feba6ffd640 10 client.4261.objecter ms_dispatch 0x7febb0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:58.655 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.677+0000 7feba6ffd640 3 client.4261.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:58.655 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.677+0000 7feba6ffd640 3 client.4261.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:58.655 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.677+0000 7feba6ffd640 20 client.4261.objecter dump_active .. 0 homeless 2024-03-29T11:21:58.660 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.684+0000 7feba6ffd640 1 -- 172.21.15.114:0/411402640 <== mon.1 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7febb01f88b0 con 0x7febb00af6c0 2024-03-29T11:21:58.660 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.684+0000 7feba6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:58.660 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:58.684+0000 7feba6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:58.990 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:59.014+0000 7febb8739640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:58.991 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:59.014+0000 7febb8739640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:58.991 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:59.014+0000 7febb8739640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:21:58.991 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:59.014+0000 7febb8739640 1 -- 172.21.15.114:0/411402640 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7febb0075f10 con 0x7febb00af6c0 2024-03-29T11:21:58.991 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:59.014+0000 7feba6ffd640 1 -- 172.21.15.114:0/411402640 <== mon.1 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2858224613) 0x7feba8050a70 con 0x7febb00af6c0 2024-03-29T11:21:58.991 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:59.015+0000 7feba6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:58.991 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:59.015+0000 7feba6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:58.992 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T11:21:58.992 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"b0de2b9b-c45a-44c7-b6ee-e83b39a1e108","created":"2024-03-29T11:20:49.420660+0000","modified":"2024-03-29T11:20:51.700167+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:51.700167+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a64785d3-6b11-499a-b706-cc827ab96a74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5882dadb-e7a8-4604-ba9a-418013c106c8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d7e8c409-bc66-4e4a-bbdb-420bad31388e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0682989-3595-4cd6-90a5-ab9111c278e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"67068120-e3ff-4817-bd06-44feb7479a35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09e8b-8a85-4997-bdea-75d754e6ee67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2f0d9ead-f47f-4cb3-92a4-0a5ad23cd831","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fdb6a25f-6bd8-4c65-9b9b-b460ba7673a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:58.995 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:59.018+0000 7febb8739640 10 librados: watch_flush enter 2024-03-29T11:21:58.995 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:59.018+0000 7febb8739640 10 librados: watch_flush exit 2024-03-29T11:21:58.995 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:59.018+0000 7febb8739640 20 client.4261.objecter shutdown clearing up homeless session... 2024-03-29T11:21:58.995 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:59.018+0000 7febb8739640 10 client.4261.objecter successfully canceled tick 2024-03-29T11:21:58.995 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:59.018+0000 7febb8739640 1 -- 172.21.15.114:0/411402640 >> v1:172.21.15.114:6816/4038755055 conn(0x7feb9406de70 legacy=0x7feb94070330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:58.995 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:59.018+0000 7febb8739640 10 monclient: shutdown 2024-03-29T11:21:58.995 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:59.018+0000 7febb8739640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:58.995 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:59.018+0000 7febb8739640 1 -- 172.21.15.114:0/411402640 >> v1:172.21.15.176:6789/0 conn(0x7febb00af6c0 legacy=0x7febb007bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:58.995 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:59.018+0000 7febb8739640 1 -- 172.21.15.114:0/411402640 shutdown_connections 2024-03-29T11:21:58.995 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:59.018+0000 7febb8739640 1 -- 172.21.15.114:0/411402640 >> 172.21.15.114:0/411402640 conn(0x7febb00b35e0 msgr2=0x7febb00b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:58.996 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:59.019+0000 7febb8739640 1 -- 172.21.15.114:0/411402640 shutdown_connections 2024-03-29T11:21:58.996 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:59.019+0000 7febb8739640 1 -- 172.21.15.114:0/411402640 wait complete. 2024-03-29T11:21:58.996 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:21:59.019+0000 7febb8739640 1 librados: shutdown 2024-03-29T11:21:59.010 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:05.816 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:05.970 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:05.993+0000 7f5519633640 10 monclient: get_monmap_and_config 2024-03-29T11:22:05.970 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:05.993+0000 7f5519633640 10 monclient: build_initial_monmap 2024-03-29T11:22:05.970 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:05.994+0000 7f5519633640 10 monclient: monmap: 2024-03-29T11:22:05.970 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:22:05.971 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:22:05.971 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:22:05.995075+0000 2024-03-29T11:22:05.971 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:22:05.995075+0000 2024-03-29T11:22:05.971 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:05.971 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:22:05.971 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:22:05.971 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T11:22:05.971 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T11:22:05.971 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:22:05.972 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:05.995+0000 7f5519633640 1 Processor -- start 2024-03-29T11:22:05.972 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:05.995+0000 7f5519633640 1 -- start start 2024-03-29T11:22:05.972 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:05.995+0000 7f5519633640 10 monclient: init 2024-03-29T11:22:05.973 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:05.996+0000 7f5519633640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:05.973 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:05.997+0000 7f5519633640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:22:05.974 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:05.997+0000 7f5519633640 10 monclient(hunting): picked mon.noname-c con 0x7f55141422b0 addr v1:172.21.15.176:6789/0 2024-03-29T11:22:05.974 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:05.997+0000 7f5519633640 10 monclient(hunting): picked mon.noname-b con 0x7f5514142690 addr v1:172.21.15.114:6790/0 2024-03-29T11:22:05.974 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:05.997+0000 7f5512ffd640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f55141422b0 0x7f551413f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.114:59882/0 (socket says 172.21.15.114:59882) 2024-03-29T11:22:05.974 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:05.997+0000 7f5512ffd640 1 -- 172.21.15.114:0/1649647449 learned_addr learned my addr 172.21.15.114:0/1649647449 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:22:05.974 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:05.997+0000 7f5519633640 10 monclient(hunting): picked mon.noname-a con 0x7f55140b17d0 addr v1:172.21.15.114:6789/0 2024-03-29T11:22:05.974 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:05.997+0000 7f5519633640 1 -- 172.21.15.114:0/1649647449 --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5514142c80 con 0x7f55140b17d0 2024-03-29T11:22:05.974 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:05.997+0000 7f5519633640 1 -- 172.21.15.114:0/1649647449 --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55140aae40 con 0x7f55141422b0 2024-03-29T11:22:05.974 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:05.997+0000 7f5519633640 1 -- 172.21.15.114:0/1649647449 --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f551413fbd0 con 0x7f5514142690 2024-03-29T11:22:05.974 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:05.997+0000 7f5519633640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:05.974 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:05.997+0000 7f5519633640 10 monclient(hunting): authenticate will time out at 996.179806s 2024-03-29T11:22:05.977 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:05.998+0000 7f5511ffb640 1 -- 172.21.15.114:0/1649647449 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4022550669 0 0) 0x7f55140aae40 con 0x7f55141422b0 2024-03-29T11:22:05.977 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:05.998+0000 7f5511ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:22:05.977 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:05.998+0000 7f5511ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:05.978 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:05.998+0000 7f5511ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:05.978 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:05.998+0000 7f5511ffb640 10 monclient(hunting): my global_id is 4264 2024-03-29T11:22:05.978 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:05.998+0000 7f5511ffb640 1 -- 172.21.15.114:0/1649647449 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54fc003520 con 0x7f55141422b0 2024-03-29T11:22:05.978 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:05.998+0000 7f5511ffb640 1 -- 172.21.15.114:0/1649647449 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4241746286 0 0) 0x7f5514142c80 con 0x7f55140b17d0 2024-03-29T11:22:05.978 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:05.999+0000 7f5511ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:22:05.978 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:05.999+0000 7f5511ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:05.978 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:05.999+0000 7f5511ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:05.978 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:05.999+0000 7f5511ffb640 10 monclient(hunting): my global_id is 4272 2024-03-29T11:22:05.978 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:05.999+0000 7f5511ffb640 1 -- 172.21.15.114:0/1649647449 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55140aae40 con 0x7f55140b17d0 2024-03-29T11:22:05.978 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:05.999+0000 7f5511ffb640 1 -- 172.21.15.114:0/1649647449 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4053955208 0 0) 0x7f54fc003520 con 0x7f55141422b0 2024-03-29T11:22:05.978 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:05.999+0000 7f5511ffb640 1 -- 172.21.15.114:0/1649647449 >> v1:172.21.15.114:6790/0 conn(0x7f5514142690 legacy=0x7f55140b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:05.978 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:05.999+0000 7f5511ffb640 1 -- 172.21.15.114:0/1649647449 >> v1:172.21.15.114:6789/0 conn(0x7f55140b17d0 legacy=0x7f55140b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:05.978 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:05.999+0000 7f5511ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:22:05.979 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:05.999+0000 7f5511ffb640 1 monclient: found mon.noname-c 2024-03-29T11:22:05.979 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:05.999+0000 7f5511ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:05.979 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:05.999+0000 7f5511ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.176:6789/0 2024-03-29T11:22:05.979 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:05.999+0000 7f5511ffb640 1 -- 172.21.15.114:0/1649647449 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5514151940 con 0x7f55141422b0 2024-03-29T11:22:05.979 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:05.999+0000 7f5511ffb640 10 monclient: _finish_auth 0 2024-03-29T11:22:05.979 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:05.999+0000 7f5511ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:22:05.979 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:05.999+0000 7f5511ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:05.979 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:05.999+0000 7f5511ffb640 1 -- 172.21.15.114:0/1649647449 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f5504001690 con 0x7f55141422b0 2024-03-29T11:22:05.979 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:05.999+0000 7f5511ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:05.979 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:05.999+0000 7f5511ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:22:05.979 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:05.999+0000 7f5511ffb640 10 monclient: dump: 2024-03-29T11:22:05.979 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:22:05.979 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:22:05.979 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:22:05.980 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:22:05.980 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:05.980 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:22:05.980 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:22:05.980 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:22:05.980 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:22:05.980 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:22:05.980 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:05.999+0000 7f5519633640 5 monclient: authenticate success, global_id 4264 2024-03-29T11:22:05.980 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:05.999+0000 7f5519633640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:05.980 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.000+0000 7f5511ffb640 1 -- 172.21.15.114:0/1649647449 <== mon.1 v1:172.21.15.176:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f55040018b0 con 0x7f55141422b0 2024-03-29T11:22:05.980 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.000+0000 7f5511ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:05.980 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.000+0000 7f5511ffb640 1 -- 172.21.15.114:0/1649647449 <== mon.1 v1:172.21.15.176:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f5504001bc0 con 0x7f55141422b0 2024-03-29T11:22:05.980 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.000+0000 7f5519633640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:05.980 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.000+0000 7f5519633640 4 set_mon_vals no callback set 2024-03-29T11:22:05.980 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.000+0000 7f5511ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:05.981 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.000+0000 7f5511ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:22:05.981 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.000+0000 7f5511ffb640 10 monclient: dump: 2024-03-29T11:22:05.981 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:22:05.981 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:22:05.981 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:22:05.981 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:22:05.981 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:05.981 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:22:05.981 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:22:05.981 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:22:05.981 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:22:05.981 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:22:05.981 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.001+0000 7f5519633640 10 monclient: shutdown 2024-03-29T11:22:05.981 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.001+0000 7f5519633640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:05.982 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.001+0000 7f5519633640 1 -- 172.21.15.114:0/1649647449 >> v1:172.21.15.176:6789/0 conn(0x7f55141422b0 legacy=0x7f551413f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:05.982 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.001+0000 7f5519633640 1 -- 172.21.15.114:0/1649647449 shutdown_connections 2024-03-29T11:22:05.982 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.001+0000 7f5519633640 1 -- 172.21.15.114:0/1649647449 >> 172.21.15.114:0/1649647449 conn(0x7f5514066a00 msgr2=0x7f5514066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:05.985 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.001+0000 7f5519633640 1 -- 172.21.15.114:0/1649647449 shutdown_connections 2024-03-29T11:22:05.985 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.002+0000 7f5519633640 1 -- 172.21.15.114:0/1649647449 wait complete. 2024-03-29T11:22:05.985 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.003+0000 7f5519633640 10 monclient: build_initial_monmap 2024-03-29T11:22:05.985 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.003+0000 7f5519633640 10 monclient: monmap: 2024-03-29T11:22:05.985 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:22:05.985 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:05.985 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T11:22:05.985 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T11:22:05.985 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:05.986 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:22:05.986 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:22:05.986 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T11:22:05.986 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T11:22:05.986 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:22:05.986 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.003+0000 7f5519633640 1 librados: starting msgr at 2024-03-29T11:22:05.986 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.003+0000 7f5519633640 1 librados: starting objecter 2024-03-29T11:22:05.986 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.004+0000 7f5519633640 1 Processor -- start 2024-03-29T11:22:05.986 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.004+0000 7f5519633640 1 -- start start 2024-03-29T11:22:05.986 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.004+0000 7f5519633640 1 librados: setting wanted keys 2024-03-29T11:22:05.986 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.004+0000 7f5519633640 1 librados: calling monclient init 2024-03-29T11:22:05.986 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.004+0000 7f5519633640 10 monclient: init 2024-03-29T11:22:05.986 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.005+0000 7f5519633640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:05.986 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.005+0000 7f5519633640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:22:05.987 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.005+0000 7f5519633640 10 monclient(hunting): picked mon.noname-c con 0x7f55140b17d0 addr v1:172.21.15.114:6790/0 2024-03-29T11:22:05.987 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.005+0000 7f5519633640 10 monclient(hunting): picked mon.noname-b con 0x7f55141422b0 addr v1:172.21.15.176:6789/0 2024-03-29T11:22:05.987 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.005+0000 7f5519633640 10 monclient(hunting): picked mon.noname-a con 0x7f5514142690 addr v1:172.21.15.114:6789/0 2024-03-29T11:22:05.987 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.005+0000 7f5519633640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55141db8c0 con 0x7f5514142690 2024-03-29T11:22:05.987 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.005+0000 7f55137fe640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f5514142690 0x7f55141e1d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:35188/0 (socket says 172.21.15.114:35188) 2024-03-29T11:22:05.987 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.005+0000 7f55137fe640 1 -- 172.21.15.114:0/2025170877 learned_addr learned my addr 172.21.15.114:0/2025170877 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:22:05.987 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.005+0000 7f5519633640 1 -- 172.21.15.114:0/2025170877 --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55140b0560 con 0x7f55141422b0 2024-03-29T11:22:05.987 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.005+0000 7f5519633640 1 -- 172.21.15.114:0/2025170877 --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55141db3a0 con 0x7f55140b17d0 2024-03-29T11:22:05.987 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.005+0000 7f54fb7fe640 10 client.?.objecter ms_handle_connect 0x7f5514142690 2024-03-29T11:22:05.987 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.005+0000 7f54fb7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:05.987 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.005+0000 7f54fb7fe640 10 client.?.objecter ms_handle_connect 0x7f55141422b0 2024-03-29T11:22:05.987 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.005+0000 7f54fb7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:05.987 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.005+0000 7f5519633640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:05.988 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.005+0000 7f5519633640 10 monclient(hunting): authenticate will time out at 996.187796s 2024-03-29T11:22:05.988 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.005+0000 7f54fb7fe640 10 client.?.objecter ms_handle_connect 0x7f55140b17d0 2024-03-29T11:22:05.988 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.005+0000 7f54fb7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:05.988 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.005+0000 7f54fb7fe640 1 -- 172.21.15.114:0/2025170877 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4198356936 0 0) 0x7f55141db8c0 con 0x7f5514142690 2024-03-29T11:22:05.988 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.005+0000 7f54fb7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:22:05.988 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.005+0000 7f54fb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:05.988 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.005+0000 7f54fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:05.988 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.005+0000 7f54fb7fe640 10 monclient(hunting): my global_id is 4275 2024-03-29T11:22:05.988 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.005+0000 7f54fb7fe640 1 -- 172.21.15.114:0/2025170877 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5500004eb0 con 0x7f5514142690 2024-03-29T11:22:05.988 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.006+0000 7f54fb7fe640 1 -- 172.21.15.114:0/2025170877 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1925451668 0 0) 0x7f55140b0560 con 0x7f55141422b0 2024-03-29T11:22:05.988 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.006+0000 7f54fb7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:22:05.988 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.006+0000 7f54fb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:05.988 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.006+0000 7f54fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:05.988 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.006+0000 7f54fb7fe640 10 monclient(hunting): my global_id is 4267 2024-03-29T11:22:05.988 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.006+0000 7f54fb7fe640 1 -- 172.21.15.114:0/2025170877 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55141db8c0 con 0x7f55141422b0 2024-03-29T11:22:05.989 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.006+0000 7f54fb7fe640 1 -- 172.21.15.114:0/2025170877 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1567135628 0 0) 0x7f55141db3a0 con 0x7f55140b17d0 2024-03-29T11:22:05.989 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.006+0000 7f54fb7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:22:05.989 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.006+0000 7f54fb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:05.989 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.006+0000 7f54fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:05.989 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.006+0000 7f54fb7fe640 10 monclient(hunting): my global_id is 4268 2024-03-29T11:22:05.989 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.006+0000 7f54fb7fe640 1 -- 172.21.15.114:0/2025170877 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55140b0560 con 0x7f55140b17d0 2024-03-29T11:22:05.989 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.006+0000 7f54fb7fe640 1 -- 172.21.15.114:0/2025170877 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4223854196 0 0) 0x7f55141db8c0 con 0x7f55141422b0 2024-03-29T11:22:05.989 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.006+0000 7f54fb7fe640 1 -- 172.21.15.114:0/2025170877 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5500005620 con 0x7f55141422b0 2024-03-29T11:22:05.989 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.006+0000 7f54fb7fe640 1 -- 172.21.15.114:0/2025170877 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f5508001810 con 0x7f55141422b0 2024-03-29T11:22:05.989 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.006+0000 7f54fb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:05.989 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.006+0000 7f54fb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:22:05.989 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.006+0000 7f54fb7fe640 10 monclient(hunting): dump: 2024-03-29T11:22:05.989 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:22:05.989 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:22:05.990 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:22:05.990 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:22:05.990 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:05.990 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:22:05.990 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:22:05.990 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:22:05.990 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:22:05.990 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:22:05.990 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.006+0000 7f54fb7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:05.990 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.007+0000 7f54fb7fe640 1 -- 172.21.15.114:0/2025170877 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1136921327 0 0) 0x7f5500004eb0 con 0x7f5514142690 2024-03-29T11:22:05.990 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.007+0000 7f54fb7fe640 1 -- 172.21.15.114:0/2025170877 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f55141db8c0 con 0x7f5514142690 2024-03-29T11:22:05.990 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.007+0000 7f54fb7fe640 1 -- 172.21.15.114:0/2025170877 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f550c001d10 con 0x7f5514142690 2024-03-29T11:22:05.990 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.007+0000 7f54fb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:05.990 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.007+0000 7f54fb7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:22:05.991 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.007+0000 7f54fb7fe640 10 monclient(hunting): dump: 2024-03-29T11:22:05.991 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:22:05.991 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:22:05.991 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:22:05.991 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:22:05.991 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:05.991 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:22:05.991 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:22:05.991 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:22:05.991 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:22:05.991 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:22:05.991 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.007+0000 7f54fb7fe640 1 -- 172.21.15.114:0/2025170877 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3317131022 0 0) 0x7f55140b0560 con 0x7f55140b17d0 2024-03-29T11:22:05.992 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.007+0000 7f54fb7fe640 1 -- 172.21.15.114:0/2025170877 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5500004eb0 con 0x7f55140b17d0 2024-03-29T11:22:05.992 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.007+0000 7f54fb7fe640 1 -- 172.21.15.114:0/2025170877 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f5504001550 con 0x7f55140b17d0 2024-03-29T11:22:05.992 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.007+0000 7f54fb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:05.992 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.007+0000 7f54fb7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:22:05.992 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.007+0000 7f54fb7fe640 10 monclient(hunting): dump: 2024-03-29T11:22:05.992 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:22:05.992 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:22:05.992 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:22:05.992 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:22:05.992 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:05.992 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:22:05.992 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:22:05.992 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:22:05.992 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:22:05.992 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:22:05.993 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.007+0000 7f54fb7fe640 1 -- 172.21.15.114:0/2025170877 <== mon.1 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3980254330 0 0) 0x7f5500005620 con 0x7f55141422b0 2024-03-29T11:22:05.993 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.007+0000 7f54fb7fe640 1 -- 172.21.15.114:0/2025170877 >> v1:172.21.15.114:6790/0 conn(0x7f55140b17d0 legacy=0x7f55140b0980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:05.993 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.007+0000 7f54fb7fe640 1 -- 172.21.15.114:0/2025170877 >> v1:172.21.15.114:6789/0 conn(0x7f5514142690 legacy=0x7f55141e1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:05.993 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.007+0000 7f54fb7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:22:05.993 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.007+0000 7f54fb7fe640 1 monclient: found mon.b 2024-03-29T11:22:05.993 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.007+0000 7f54fb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:05.993 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.007+0000 7f54fb7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:22:05.993 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.007+0000 7f54fb7fe640 1 -- 172.21.15.114:0/2025170877 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55141daee0 con 0x7f55141422b0 2024-03-29T11:22:05.993 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.007+0000 7f54fb7fe640 10 monclient: _finish_auth 0 2024-03-29T11:22:05.993 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.007+0000 7f54fb7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:22:05.993 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.007+0000 7f54fb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:05.993 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.008+0000 7f5519633640 5 monclient: authenticate success, global_id 4267 2024-03-29T11:22:05.993 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.008+0000 7f5519633640 10 monclient: _renew_subs 2024-03-29T11:22:05.993 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.008+0000 7f5519633640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:22:05.994 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.008+0000 7f5519633640 1 -- 172.21.15.114:0/2025170877 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55141db130 con 0x7f55141422b0 2024-03-29T11:22:05.994 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.008+0000 7f5519633640 10 client.4267.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:05.994 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.008+0000 7f5519633640 10 monclient: _renew_subs 2024-03-29T11:22:05.994 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.008+0000 7f5519633640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:22:05.994 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.008+0000 7f5519633640 1 -- 172.21.15.114:0/2025170877 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f55140b1270 con 0x7f55141422b0 2024-03-29T11:22:05.996 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.008+0000 7f5519633640 1 librados: init done 2024-03-29T11:22:05.996 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.008+0000 7f54fb7fe640 1 -- 172.21.15.114:0/2025170877 <== mon.1 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5508001d00 con 0x7f55141422b0 2024-03-29T11:22:05.996 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.008+0000 7f54fb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:05.996 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.008+0000 7f55117fa640 4 set_mon_vals no callback set 2024-03-29T11:22:05.996 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.008+0000 7f54fb7fe640 1 -- 172.21.15.114:0/2025170877 <== mon.1 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f5508003790 con 0x7f55141422b0 2024-03-29T11:22:05.996 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.008+0000 7f54fb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:05.996 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.008+0000 7f54fb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:22:05.996 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.008+0000 7f54fb7fe640 10 monclient: dump: 2024-03-29T11:22:05.996 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:22:05.996 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:22:05.996 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:22:05.996 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:22:05.997 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:05.997 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:22:05.997 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:22:05.997 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:22:05.997 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:22:05.997 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:22:05.997 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.009+0000 7f5519633640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:05.997 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.009+0000 7f5519633640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:05.997 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.009+0000 7f5519633640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:22:05.997 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.009+0000 7f5519633640 1 -- 172.21.15.114:0/2025170877 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54f4005080 con 0x7f55141422b0 2024-03-29T11:22:05.997 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.011+0000 7f54fb7fe640 1 -- 172.21.15.114:0/2025170877 <== mon.1 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1594292461 0 0) 0x7f5508003970 con 0x7f55141422b0 2024-03-29T11:22:05.997 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.012+0000 7f54fb7fe640 1 -- 172.21.15.114:0/2025170877 <== mon.1 v1:172.21.15.176:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3238987313 0 0) 0x7f5508083800 con 0x7f55141422b0 2024-03-29T11:22:05.997 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.012+0000 7f54fb7fe640 10 client.4267.objecter ms_dispatch 0x7f5514000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:22:05.997 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.012+0000 7f54fb7fe640 3 client.4267.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:22:05.998 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.012+0000 7f54fb7fe640 3 client.4267.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:22:05.998 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.012+0000 7f54fb7fe640 20 client.4267.objecter dump_active .. 0 homeless 2024-03-29T11:22:05.998 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.012+0000 7f5512ffd640 10 monclient: get_auth_request con 0x7f550006e260 auth_method 0 2024-03-29T11:22:05.998 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.012+0000 7f54fb7fe640 10 client.4267.objecter ms_handle_connect 0x7f550006e260 2024-03-29T11:22:05.998 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.017+0000 7f54fb7fe640 1 -- 172.21.15.114:0/2025170877 <== mon.1 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f550804f0e0 con 0x7f55141422b0 2024-03-29T11:22:05.998 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.017+0000 7f54fb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:05.998 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.017+0000 7f54fb7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:06.396 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.419+0000 7f5519633640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:06.396 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.419+0000 7f5519633640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:06.396 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.419+0000 7f5519633640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:22:06.396 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.419+0000 7f5519633640 1 -- 172.21.15.114:0/2025170877 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f54f4005330 con 0x7f55141422b0 2024-03-29T11:22:06.397 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.420+0000 7f54fb7fe640 1 -- 172.21.15.114:0/2025170877 <== mon.1 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2858224613) 0x7f5508051010 con 0x7f55141422b0 2024-03-29T11:22:06.397 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.420+0000 7f54fb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:06.397 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.420+0000 7f54fb7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:06.397 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T11:22:06.397 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"b0de2b9b-c45a-44c7-b6ee-e83b39a1e108","created":"2024-03-29T11:20:49.420660+0000","modified":"2024-03-29T11:20:51.700167+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:51.700167+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a64785d3-6b11-499a-b706-cc827ab96a74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5882dadb-e7a8-4604-ba9a-418013c106c8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d7e8c409-bc66-4e4a-bbdb-420bad31388e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0682989-3595-4cd6-90a5-ab9111c278e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"67068120-e3ff-4817-bd06-44feb7479a35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09e8b-8a85-4997-bdea-75d754e6ee67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2f0d9ead-f47f-4cb3-92a4-0a5ad23cd831","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fdb6a25f-6bd8-4c65-9b9b-b460ba7673a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:06.401 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.424+0000 7f5519633640 10 librados: watch_flush enter 2024-03-29T11:22:06.402 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.424+0000 7f5519633640 10 librados: watch_flush exit 2024-03-29T11:22:06.402 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.425+0000 7f5519633640 20 client.4267.objecter shutdown clearing up homeless session... 2024-03-29T11:22:06.402 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.425+0000 7f5519633640 10 client.4267.objecter successfully canceled tick 2024-03-29T11:22:06.402 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.425+0000 7f5519633640 1 -- 172.21.15.114:0/2025170877 >> v1:172.21.15.114:6816/4038755055 conn(0x7f550006e260 legacy=0x7f5500070720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:06.402 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.425+0000 7f5519633640 10 monclient: shutdown 2024-03-29T11:22:06.402 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.425+0000 7f5519633640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:06.402 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.425+0000 7f5519633640 1 -- 172.21.15.114:0/2025170877 >> v1:172.21.15.176:6789/0 conn(0x7f55141422b0 legacy=0x7f55140adf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:06.402 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.425+0000 7f5519633640 1 -- 172.21.15.114:0/2025170877 shutdown_connections 2024-03-29T11:22:06.402 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.425+0000 7f5519633640 1 -- 172.21.15.114:0/2025170877 >> 172.21.15.114:0/2025170877 conn(0x7f5514066a00 msgr2=0x7f551413f290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:06.402 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.425+0000 7f5519633640 1 -- 172.21.15.114:0/2025170877 shutdown_connections 2024-03-29T11:22:06.402 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.425+0000 7f5519633640 1 -- 172.21.15.114:0/2025170877 wait complete. 2024-03-29T11:22:06.402 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:06.425+0000 7f5519633640 1 librados: shutdown 2024-03-29T11:22:06.417 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:13.224 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:13.361 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.384+0000 7fd8c0d02640 10 monclient: get_monmap_and_config 2024-03-29T11:22:13.361 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.384+0000 7fd8c0d02640 10 monclient: build_initial_monmap 2024-03-29T11:22:13.361 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.384+0000 7fd8c0d02640 10 monclient: monmap: 2024-03-29T11:22:13.361 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:22:13.361 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:22:13.362 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:22:13.385621+0000 2024-03-29T11:22:13.362 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:22:13.385621+0000 2024-03-29T11:22:13.362 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:13.362 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:22:13.362 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:22:13.362 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T11:22:13.362 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T11:22:13.362 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:22:13.362 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.385+0000 7fd8c0d02640 1 Processor -- start 2024-03-29T11:22:13.364 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.385+0000 7fd8c0d02640 1 -- start start 2024-03-29T11:22:13.364 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.385+0000 7fd8c0d02640 10 monclient: init 2024-03-29T11:22:13.364 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.386+0000 7fd8c0d02640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:13.364 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.386+0000 7fd8c0d02640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:22:13.364 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.386+0000 7fd8c0d02640 10 monclient(hunting): picked mon.noname-b con 0x7fd8bc1457f0 addr v1:172.21.15.114:6790/0 2024-03-29T11:22:13.364 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.386+0000 7fd8c0d02640 10 monclient(hunting): picked mon.noname-c con 0x7fd8bc13f650 addr v1:172.21.15.176:6789/0 2024-03-29T11:22:13.364 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.386+0000 7fd8c0d02640 10 monclient(hunting): picked mon.noname-a con 0x7fd8bc0a9d70 addr v1:172.21.15.114:6789/0 2024-03-29T11:22:13.364 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.386+0000 7fd8c0d02640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8bc0aaef0 con 0x7fd8bc0a9d70 2024-03-29T11:22:13.364 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.386+0000 7fd8c0d02640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8bc145bd0 con 0x7fd8bc13f650 2024-03-29T11:22:13.364 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.386+0000 7fd8c0d02640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8bc145eb0 con 0x7fd8bc1457f0 2024-03-29T11:22:13.365 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.386+0000 7fd8c0d02640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:13.365 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.386+0000 7fd8c0d02640 10 monclient(hunting): authenticate will time out at 1003.568901s 2024-03-29T11:22:13.365 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.387+0000 7fd8b9d9b640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7fd8bc13f650 0x7fd8bc0b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.114:51594/0 (socket says 172.21.15.114:51594) 2024-03-29T11:22:13.365 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.387+0000 7fd8b9d9b640 1 -- 172.21.15.114:0/3485340391 learned_addr learned my addr 172.21.15.114:0/3485340391 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:22:13.366 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.388+0000 7fd8b959a640 1 -- 172.21.15.114:0/3485340391 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1392876964 0 0) 0x7fd8bc145eb0 con 0x7fd8bc1457f0 2024-03-29T11:22:13.366 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.388+0000 7fd8b959a640 10 monclient(hunting): _negotiate 2024-03-29T11:22:13.367 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.388+0000 7fd8b959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:13.367 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.388+0000 7fd8b959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:13.367 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.388+0000 7fd8b959a640 10 monclient(hunting): my global_id is 4271 2024-03-29T11:22:13.367 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.388+0000 7fd8b959a640 1 -- 172.21.15.114:0/3485340391 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8a4003520 con 0x7fd8bc1457f0 2024-03-29T11:22:13.367 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.388+0000 7fd8b959a640 1 -- 172.21.15.114:0/3485340391 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 677396252 0 0) 0x7fd8a4003520 con 0x7fd8bc1457f0 2024-03-29T11:22:13.367 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.388+0000 7fd8b959a640 1 -- 172.21.15.114:0/3485340391 >> v1:172.21.15.176:6789/0 conn(0x7fd8bc13f650 legacy=0x7fd8bc0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:22:13.367 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.388+0000 7fd8b959a640 1 -- 172.21.15.114:0/3485340391 >> v1:172.21.15.114:6789/0 conn(0x7fd8bc0a9d70 legacy=0x7fd8bc14d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:22:13.367 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.389+0000 7fd8b959a640 10 monclient: _finish_hunting 0 2024-03-29T11:22:13.368 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.389+0000 7fd8b959a640 1 monclient: found mon.noname-b 2024-03-29T11:22:13.368 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.389+0000 7fd8b959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:13.368 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.389+0000 7fd8b959a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.114:6790/0 2024-03-29T11:22:13.368 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.389+0000 7fd8b959a640 1 -- 172.21.15.114:0/3485340391 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8bc146340 con 0x7fd8bc1457f0 2024-03-29T11:22:13.368 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.389+0000 7fd8b959a640 10 monclient: _finish_auth 0 2024-03-29T11:22:13.368 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.389+0000 7fd8b959a640 10 monclient: _check_auth_tickets 2024-03-29T11:22:13.368 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.389+0000 7fd8b959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:13.368 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.389+0000 7fd8b959a640 1 -- 172.21.15.114:0/3485340391 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fd8ac001690 con 0x7fd8bc1457f0 2024-03-29T11:22:13.368 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.389+0000 7fd8b959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:13.368 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.389+0000 7fd8b959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:22:13.368 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.389+0000 7fd8b959a640 10 monclient: dump: 2024-03-29T11:22:13.368 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:22:13.368 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:22:13.368 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:22:13.368 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:22:13.369 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:13.369 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:22:13.369 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:22:13.369 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:22:13.369 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:22:13.369 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:22:13.369 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.389+0000 7fd8c0d02640 5 monclient: authenticate success, global_id 4271 2024-03-29T11:22:13.369 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.389+0000 7fd8c0d02640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:13.369 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.389+0000 7fd8b959a640 1 -- 172.21.15.114:0/3485340391 <== mon.2 v1:172.21.15.114:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd8ac001860 con 0x7fd8bc1457f0 2024-03-29T11:22:13.369 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.389+0000 7fd8b959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:13.369 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.389+0000 7fd8b959a640 1 -- 172.21.15.114:0/3485340391 <== mon.2 v1:172.21.15.114:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fd8ac001cb0 con 0x7fd8bc1457f0 2024-03-29T11:22:13.369 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.389+0000 7fd8c0d02640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:13.369 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.389+0000 7fd8c0d02640 4 set_mon_vals no callback set 2024-03-29T11:22:13.370 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.389+0000 7fd8b959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:13.370 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.389+0000 7fd8b959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:22:13.370 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.389+0000 7fd8b959a640 10 monclient: dump: 2024-03-29T11:22:13.370 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:22:13.370 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:22:13.370 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:22:13.370 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:22:13.370 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:13.370 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:22:13.370 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:22:13.370 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:22:13.370 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:22:13.370 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:22:13.377 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.390+0000 7fd8c0d02640 10 monclient: shutdown 2024-03-29T11:22:13.377 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.390+0000 7fd8c0d02640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:13.377 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.390+0000 7fd8c0d02640 1 -- 172.21.15.114:0/3485340391 >> v1:172.21.15.114:6790/0 conn(0x7fd8bc1457f0 legacy=0x7fd8bc13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:13.377 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.390+0000 7fd8c0d02640 1 -- 172.21.15.114:0/3485340391 shutdown_connections 2024-03-29T11:22:13.377 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.390+0000 7fd8c0d02640 1 -- 172.21.15.114:0/3485340391 >> 172.21.15.114:0/3485340391 conn(0x7fd8bc066a00 msgr2=0x7fd8bc066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:13.377 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.390+0000 7fd8c0d02640 1 -- 172.21.15.114:0/3485340391 shutdown_connections 2024-03-29T11:22:13.377 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.390+0000 7fd8c0d02640 1 -- 172.21.15.114:0/3485340391 wait complete. 2024-03-29T11:22:13.377 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.391+0000 7fd8c0d02640 10 monclient: build_initial_monmap 2024-03-29T11:22:13.377 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.391+0000 7fd8c0d02640 10 monclient: monmap: 2024-03-29T11:22:13.377 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:22:13.377 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:13.377 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T11:22:13.378 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T11:22:13.378 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:13.378 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:22:13.378 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:22:13.378 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T11:22:13.378 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T11:22:13.378 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:22:13.378 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.393+0000 7fd8c0d02640 1 librados: starting msgr at 2024-03-29T11:22:13.378 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.393+0000 7fd8c0d02640 1 librados: starting objecter 2024-03-29T11:22:13.378 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.393+0000 7fd8c0d02640 1 Processor -- start 2024-03-29T11:22:13.378 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.393+0000 7fd8c0d02640 1 -- start start 2024-03-29T11:22:13.378 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.393+0000 7fd8c0d02640 1 librados: setting wanted keys 2024-03-29T11:22:13.378 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.393+0000 7fd8c0d02640 1 librados: calling monclient init 2024-03-29T11:22:13.379 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.393+0000 7fd8c0d02640 10 monclient: init 2024-03-29T11:22:13.379 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.394+0000 7fd8c0d02640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:13.379 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.394+0000 7fd8c0d02640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:22:13.379 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.394+0000 7fd8c0d02640 10 monclient(hunting): picked mon.noname-b con 0x7fd8bc0a9d70 addr v1:172.21.15.176:6789/0 2024-03-29T11:22:13.379 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.394+0000 7fd8c0d02640 10 monclient(hunting): picked mon.noname-a con 0x7fd8bc13f650 addr v1:172.21.15.114:6789/0 2024-03-29T11:22:13.379 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.394+0000 7fd8c0d02640 10 monclient(hunting): picked mon.noname-c con 0x7fd8bc1457f0 addr v1:172.21.15.114:6790/0 2024-03-29T11:22:13.379 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.394+0000 7fd8c0d02640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8bc13a3b0 con 0x7fd8bc13f650 2024-03-29T11:22:13.379 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.394+0000 7fd8c0d02640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8bc141ee0 con 0x7fd8bc0a9d70 2024-03-29T11:22:13.379 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.394+0000 7fd8c0d02640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8bc142070 con 0x7fd8bc1457f0 2024-03-29T11:22:13.379 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.394+0000 7fd8c0d02640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:13.379 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.394+0000 7fd8c0d02640 10 monclient(hunting): authenticate will time out at 1003.577001s 2024-03-29T11:22:13.379 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.395+0000 7fd8ba59c640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7fd8bc0a9d70 0x7fd8bc136cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.114:51784/0 (socket says 172.21.15.114:51784) 2024-03-29T11:22:13.379 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.395+0000 7fd8ba59c640 1 -- 172.21.15.114:0/2316054251 learned_addr learned my addr 172.21.15.114:0/2316054251 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:22:13.379 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.395+0000 7fd8aaffd640 10 client.?.objecter ms_handle_connect 0x7fd8bc0a9d70 2024-03-29T11:22:13.380 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.395+0000 7fd8aaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:13.380 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.395+0000 7fd8aaffd640 1 -- 172.21.15.114:0/2316054251 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3106777067 0 0) 0x7fd8bc141ee0 con 0x7fd8bc0a9d70 2024-03-29T11:22:13.380 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.395+0000 7fd8aaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:22:13.380 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.395+0000 7fd8aaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:13.380 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.395+0000 7fd8aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:13.380 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.395+0000 7fd8aaffd640 10 monclient(hunting): my global_id is 4270 2024-03-29T11:22:13.380 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.395+0000 7fd8aaffd640 1 -- 172.21.15.114:0/2316054251 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd89c004eb0 con 0x7fd8bc0a9d70 2024-03-29T11:22:13.380 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.395+0000 7fd8aaffd640 10 client.?.objecter ms_handle_connect 0x7fd8bc1457f0 2024-03-29T11:22:13.380 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.395+0000 7fd8aaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:13.380 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.395+0000 7fd8aaffd640 1 -- 172.21.15.114:0/2316054251 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3835789042 0 0) 0x7fd8bc142070 con 0x7fd8bc1457f0 2024-03-29T11:22:13.380 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.395+0000 7fd8aaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:22:13.380 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.395+0000 7fd8aaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:13.380 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.395+0000 7fd8aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:13.380 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.395+0000 7fd8aaffd640 10 monclient(hunting): my global_id is 4274 2024-03-29T11:22:13.380 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.395+0000 7fd8aaffd640 1 -- 172.21.15.114:0/2316054251 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8bc141ee0 con 0x7fd8bc1457f0 2024-03-29T11:22:13.381 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.396+0000 7fd8aaffd640 1 -- 172.21.15.114:0/2316054251 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3357689688 0 0) 0x7fd8bc141ee0 con 0x7fd8bc1457f0 2024-03-29T11:22:13.381 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.396+0000 7fd8aaffd640 1 -- 172.21.15.114:0/2316054251 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd8bc142070 con 0x7fd8bc1457f0 2024-03-29T11:22:13.381 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.396+0000 7fd8aaffd640 1 -- 172.21.15.114:0/2316054251 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fd8b4001b30 con 0x7fd8bc1457f0 2024-03-29T11:22:13.381 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.396+0000 7fd8aaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:13.381 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.396+0000 7fd8aaffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:22:13.381 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.396+0000 7fd8aaffd640 10 monclient(hunting): dump: 2024-03-29T11:22:13.381 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:22:13.381 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:22:13.381 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:22:13.381 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:22:13.381 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:13.381 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:22:13.381 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:22:13.381 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:22:13.382 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:22:13.382 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:22:13.382 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.396+0000 7fd8aaffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:13.382 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.397+0000 7fd8aaffd640 10 client.?.objecter ms_handle_connect 0x7fd8bc13f650 2024-03-29T11:22:13.382 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.397+0000 7fd8aaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:13.382 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.397+0000 7fd8aaffd640 1 -- 172.21.15.114:0/2316054251 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2806796920 0 0) 0x7fd89c004eb0 con 0x7fd8bc0a9d70 2024-03-29T11:22:13.382 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.397+0000 7fd8aaffd640 1 -- 172.21.15.114:0/2316054251 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd8bc141ee0 con 0x7fd8bc0a9d70 2024-03-29T11:22:13.382 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.397+0000 7fd8aaffd640 1 -- 172.21.15.114:0/2316054251 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fd8ac001c30 con 0x7fd8bc0a9d70 2024-03-29T11:22:13.382 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.397+0000 7fd8aaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:13.382 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.397+0000 7fd8aaffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:22:13.382 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.397+0000 7fd8aaffd640 10 monclient(hunting): dump: 2024-03-29T11:22:13.382 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:22:13.382 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:22:13.382 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:22:13.383 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:22:13.383 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:13.383 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:22:13.383 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:22:13.383 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:22:13.383 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:22:13.383 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:22:13.383 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.397+0000 7fd8aaffd640 1 -- 172.21.15.114:0/2316054251 <== mon.2 v1:172.21.15.114:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2003322720 0 0) 0x7fd8bc142070 con 0x7fd8bc1457f0 2024-03-29T11:22:13.383 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.397+0000 7fd8aaffd640 1 -- 172.21.15.114:0/2316054251 >> v1:172.21.15.176:6789/0 conn(0x7fd8bc0a9d70 legacy=0x7fd8bc136cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:13.383 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.397+0000 7fd8aaffd640 1 -- 172.21.15.114:0/2316054251 >> v1:172.21.15.114:6789/0 conn(0x7fd8bc13f650 legacy=0x7fd8bc1d2b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:13.383 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.397+0000 7fd8aaffd640 10 monclient: _finish_hunting 0 2024-03-29T11:22:13.383 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.397+0000 7fd8aaffd640 1 monclient: found mon.c 2024-03-29T11:22:13.383 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.397+0000 7fd8aaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:13.383 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.397+0000 7fd8aaffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:22:13.384 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.397+0000 7fd8aaffd640 1 -- 172.21.15.114:0/2316054251 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8bc142200 con 0x7fd8bc1457f0 2024-03-29T11:22:13.384 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.397+0000 7fd8aaffd640 10 monclient: _finish_auth 0 2024-03-29T11:22:13.384 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.397+0000 7fd8aaffd640 10 monclient: _check_auth_tickets 2024-03-29T11:22:13.384 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.397+0000 7fd8aaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:13.384 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.397+0000 7fd8aaffd640 1 -- 172.21.15.114:0/2316054251 <== mon.2 v1:172.21.15.114:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd8b4002040 con 0x7fd8bc1457f0 2024-03-29T11:22:13.384 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.397+0000 7fd8aaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:13.384 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.397+0000 7fd8aaffd640 1 -- 172.21.15.114:0/2316054251 <== mon.2 v1:172.21.15.114:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fd8b4003b10 con 0x7fd8bc1457f0 2024-03-29T11:22:13.384 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.397+0000 7fd8aaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:13.384 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.397+0000 7fd8aaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:22:13.384 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.397+0000 7fd8aaffd640 10 monclient: dump: 2024-03-29T11:22:13.384 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:22:13.384 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:22:13.384 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:22:13.384 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:22:13.385 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:13.385 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:22:13.385 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:22:13.385 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:22:13.385 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:22:13.385 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:22:13.385 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.398+0000 7fd8b8d99640 4 set_mon_vals no callback set 2024-03-29T11:22:13.385 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.398+0000 7fd8c0d02640 5 monclient: authenticate success, global_id 4274 2024-03-29T11:22:13.385 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.398+0000 7fd8c0d02640 10 monclient: _renew_subs 2024-03-29T11:22:13.385 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.398+0000 7fd8c0d02640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:22:13.385 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.398+0000 7fd8c0d02640 1 -- 172.21.15.114:0/2316054251 --> v1:172.21.15.114:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8bc139d90 con 0x7fd8bc1457f0 2024-03-29T11:22:13.385 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.398+0000 7fd8c0d02640 10 client.4274.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:13.385 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.398+0000 7fd8c0d02640 10 monclient: _renew_subs 2024-03-29T11:22:13.385 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.398+0000 7fd8c0d02640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:22:13.385 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.398+0000 7fd8c0d02640 1 -- 172.21.15.114:0/2316054251 --> v1:172.21.15.114:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8bc1373d0 con 0x7fd8bc1457f0 2024-03-29T11:22:13.386 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.398+0000 7fd8c0d02640 1 librados: init done 2024-03-29T11:22:13.386 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.400+0000 7fd8aaffd640 1 -- 172.21.15.114:0/2316054251 <== mon.2 v1:172.21.15.114:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1594292461 0 0) 0x7fd8b4019ae0 con 0x7fd8bc1457f0 2024-03-29T11:22:13.386 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.400+0000 7fd8c0d02640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:13.386 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.400+0000 7fd8c0d02640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:13.386 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.400+0000 7fd8c0d02640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:22:13.386 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.400+0000 7fd8c0d02640 1 -- 172.21.15.114:0/2316054251 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8bc13a540 con 0x7fd8bc1457f0 2024-03-29T11:22:13.387 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.401+0000 7fd8aaffd640 1 -- 172.21.15.114:0/2316054251 <== mon.2 v1:172.21.15.114:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3238987313 0 0) 0x7fd8b4083a20 con 0x7fd8bc1457f0 2024-03-29T11:22:13.387 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.401+0000 7fd8ba59c640 10 monclient: get_auth_request con 0x7fd89c06ded0 auth_method 0 2024-03-29T11:22:13.387 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.401+0000 7fd8aaffd640 10 client.4274.objecter ms_dispatch 0x7fd8bc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:22:13.387 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.401+0000 7fd8aaffd640 3 client.4274.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:22:13.387 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.401+0000 7fd8aaffd640 3 client.4274.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:22:13.387 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.401+0000 7fd8aaffd640 20 client.4274.objecter dump_active .. 0 homeless 2024-03-29T11:22:13.387 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.401+0000 7fd8aaffd640 10 client.4274.objecter ms_handle_connect 0x7fd89c06ded0 2024-03-29T11:22:13.388 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.409+0000 7fd8aaffd640 1 -- 172.21.15.114:0/2316054251 <== mon.2 v1:172.21.15.114:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd8b404f300 con 0x7fd8bc1457f0 2024-03-29T11:22:13.388 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.409+0000 7fd8aaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:13.388 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.409+0000 7fd8aaffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:13.690 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.713+0000 7fd8c0d02640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:13.690 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.713+0000 7fd8c0d02640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:13.690 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.713+0000 7fd8c0d02640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:22:13.690 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.713+0000 7fd8c0d02640 1 -- 172.21.15.114:0/2316054251 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd8bc139f20 con 0x7fd8bc1457f0 2024-03-29T11:22:13.691 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.714+0000 7fd8aaffd640 1 -- 172.21.15.114:0/2316054251 <== mon.2 v1:172.21.15.114:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2858224613) 0x7fd8b4051230 con 0x7fd8bc1457f0 2024-03-29T11:22:13.691 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.714+0000 7fd8aaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:13.691 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.714+0000 7fd8aaffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:13.691 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T11:22:13.691 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"b0de2b9b-c45a-44c7-b6ee-e83b39a1e108","created":"2024-03-29T11:20:49.420660+0000","modified":"2024-03-29T11:20:51.700167+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:51.700167+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a64785d3-6b11-499a-b706-cc827ab96a74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5882dadb-e7a8-4604-ba9a-418013c106c8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d7e8c409-bc66-4e4a-bbdb-420bad31388e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0682989-3595-4cd6-90a5-ab9111c278e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"67068120-e3ff-4817-bd06-44feb7479a35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09e8b-8a85-4997-bdea-75d754e6ee67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2f0d9ead-f47f-4cb3-92a4-0a5ad23cd831","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fdb6a25f-6bd8-4c65-9b9b-b460ba7673a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:13.693 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.717+0000 7fd8c0d02640 10 librados: watch_flush enter 2024-03-29T11:22:13.693 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.717+0000 7fd8c0d02640 10 librados: watch_flush exit 2024-03-29T11:22:13.693 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.717+0000 7fd8c0d02640 20 client.4274.objecter shutdown clearing up homeless session... 2024-03-29T11:22:13.693 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.717+0000 7fd8c0d02640 10 client.4274.objecter successfully canceled tick 2024-03-29T11:22:13.694 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.718+0000 7fd8c0d02640 1 -- 172.21.15.114:0/2316054251 >> v1:172.21.15.114:6816/4038755055 conn(0x7fd89c06ded0 legacy=0x7fd89c070390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:13.694 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.718+0000 7fd8c0d02640 10 monclient: shutdown 2024-03-29T11:22:13.694 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.718+0000 7fd8c0d02640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:13.695 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.718+0000 7fd8c0d02640 1 -- 172.21.15.114:0/2316054251 >> v1:172.21.15.114:6790/0 conn(0x7fd8bc1457f0 legacy=0x7fd8bc1de430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:13.695 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.718+0000 7fd8c0d02640 1 -- 172.21.15.114:0/2316054251 shutdown_connections 2024-03-29T11:22:13.695 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.718+0000 7fd8c0d02640 1 -- 172.21.15.114:0/2316054251 >> 172.21.15.114:0/2316054251 conn(0x7fd8bc066a00 msgr2=0x7fd8bc13cad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:13.695 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.718+0000 7fd8c0d02640 1 -- 172.21.15.114:0/2316054251 shutdown_connections 2024-03-29T11:22:13.695 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.718+0000 7fd8c0d02640 1 -- 172.21.15.114:0/2316054251 wait complete. 2024-03-29T11:22:13.695 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:13.718+0000 7fd8c0d02640 1 librados: shutdown 2024-03-29T11:22:13.709 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:20.515 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:20.661 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.685+0000 7f2fb984a640 10 monclient: get_monmap_and_config 2024-03-29T11:22:20.661 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.685+0000 7f2fb984a640 10 monclient: build_initial_monmap 2024-03-29T11:22:20.661 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.685+0000 7f2fb984a640 10 monclient: monmap: 2024-03-29T11:22:20.661 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:22:20.661 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:22:20.662 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:22:20.686176+0000 2024-03-29T11:22:20.662 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:22:20.686176+0000 2024-03-29T11:22:20.662 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:20.662 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:22:20.662 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:22:20.662 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T11:22:20.662 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T11:22:20.662 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:22:20.664 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.687+0000 7f2fb984a640 1 Processor -- start 2024-03-29T11:22:20.664 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.687+0000 7f2fb984a640 1 -- start start 2024-03-29T11:22:20.665 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.687+0000 7f2fb984a640 10 monclient: init 2024-03-29T11:22:20.665 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.688+0000 7f2fb984a640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:20.665 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.688+0000 7f2fb984a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:22:20.665 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.688+0000 7f2fb984a640 10 monclient(hunting): picked mon.noname-b con 0x7f2fb4149b50 addr v1:172.21.15.114:6790/0 2024-03-29T11:22:20.665 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.688+0000 7f2fb984a640 10 monclient(hunting): picked mon.noname-c con 0x7f2fb40b2620 addr v1:172.21.15.176:6789/0 2024-03-29T11:22:20.665 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.688+0000 7f2fb984a640 10 monclient(hunting): picked mon.noname-a con 0x7f2fb414fc70 addr v1:172.21.15.114:6789/0 2024-03-29T11:22:20.666 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.688+0000 7f2fb984a640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fb406eee0 con 0x7f2fb414fc70 2024-03-29T11:22:20.666 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.689+0000 7f2fb984a640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fb406f1c0 con 0x7f2fb40b2620 2024-03-29T11:22:20.666 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.689+0000 7f2fb984a640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fb4071d80 con 0x7f2fb4149b50 2024-03-29T11:22:20.666 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.689+0000 7f2fb984a640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:20.666 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.689+0000 7f2fb984a640 10 monclient(hunting): authenticate will time out at 1010.871167s 2024-03-29T11:22:20.666 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.689+0000 7f2fabfff640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f2fb40b2620 0x7f2fb40b2a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.114:46162/0 (socket says 172.21.15.114:46162) 2024-03-29T11:22:20.666 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.689+0000 7f2fabfff640 1 -- 172.21.15.114:0/3258793205 learned_addr learned my addr 172.21.15.114:0/3258793205 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:22:20.667 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.691+0000 7f2fb27fc640 1 -- 172.21.15.114:0/3258793205 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1077540198 0 0) 0x7f2fb406f1c0 con 0x7f2fb40b2620 2024-03-29T11:22:20.667 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.691+0000 7f2fb27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:22:20.667 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.691+0000 7f2fb27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:20.667 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.691+0000 7f2fb27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:20.667 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.691+0000 7f2fb27fc640 10 monclient(hunting): my global_id is 4273 2024-03-29T11:22:20.667 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.691+0000 7f2fb27fc640 1 -- 172.21.15.114:0/3258793205 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fa0003520 con 0x7f2fb40b2620 2024-03-29T11:22:20.669 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.691+0000 7f2fb27fc640 1 -- 172.21.15.114:0/3258793205 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2784190318 0 0) 0x7f2fa0003520 con 0x7f2fb40b2620 2024-03-29T11:22:20.669 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.691+0000 7f2fb27fc640 1 -- 172.21.15.114:0/3258793205 >> v1:172.21.15.114:6790/0 conn(0x7f2fb4149b50 legacy=0x7f2fb4143440 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:22:20.669 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.692+0000 7f2fb27fc640 1 -- 172.21.15.114:0/3258793205 >> v1:172.21.15.114:6789/0 conn(0x7f2fb414fc70 legacy=0x7f2fb4152060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:20.669 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.692+0000 7f2fb27fc640 10 monclient: _finish_hunting 0 2024-03-29T11:22:20.669 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.692+0000 7f2fb27fc640 1 monclient: found mon.noname-c 2024-03-29T11:22:20.669 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.692+0000 7f2fb27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:20.669 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.692+0000 7f2fb27fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.176:6789/0 2024-03-29T11:22:20.670 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.692+0000 7f2fb27fc640 1 -- 172.21.15.114:0/3258793205 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fb414a590 con 0x7f2fb40b2620 2024-03-29T11:22:20.670 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.692+0000 7f2fb27fc640 10 monclient: _finish_auth 0 2024-03-29T11:22:20.670 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.692+0000 7f2fb27fc640 10 monclient: _check_auth_tickets 2024-03-29T11:22:20.670 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.692+0000 7f2fb27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:20.670 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.692+0000 7f2fb27fc640 1 -- 172.21.15.114:0/3258793205 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f2f9c001690 con 0x7f2fb40b2620 2024-03-29T11:22:20.670 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.692+0000 7f2fb27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:20.670 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.692+0000 7f2fb27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:22:20.670 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.692+0000 7f2fb27fc640 10 monclient: dump: 2024-03-29T11:22:20.670 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:22:20.670 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:22:20.670 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:22:20.670 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:22:20.670 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:20.670 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:22:20.671 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:22:20.671 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:22:20.671 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:22:20.671 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:22:20.671 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.692+0000 7f2fb984a640 5 monclient: authenticate success, global_id 4273 2024-03-29T11:22:20.671 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.692+0000 7f2fb984a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:20.672 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.693+0000 7f2fb27fc640 1 -- 172.21.15.114:0/3258793205 <== mon.1 v1:172.21.15.176:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2f9c001920 con 0x7f2fb40b2620 2024-03-29T11:22:20.672 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.693+0000 7f2fb27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:20.672 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.693+0000 7f2fb27fc640 1 -- 172.21.15.114:0/3258793205 <== mon.1 v1:172.21.15.176:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f2f9c001d30 con 0x7f2fb40b2620 2024-03-29T11:22:20.672 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.693+0000 7f2fb27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:20.672 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.693+0000 7f2fb27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:22:20.672 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.693+0000 7f2fb27fc640 10 monclient: dump: 2024-03-29T11:22:20.672 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:22:20.672 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:22:20.672 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:22:20.672 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:22:20.672 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:20.672 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:22:20.672 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:22:20.673 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:22:20.673 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:22:20.673 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:22:20.673 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.694+0000 7f2fb984a640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:20.673 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.694+0000 7f2fb984a640 4 set_mon_vals no callback set 2024-03-29T11:22:20.673 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.694+0000 7f2fb984a640 10 monclient: shutdown 2024-03-29T11:22:20.673 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.694+0000 7f2fb984a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:20.673 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.694+0000 7f2fb984a640 1 -- 172.21.15.114:0/3258793205 >> v1:172.21.15.176:6789/0 conn(0x7f2fb40b2620 legacy=0x7f2fb40b2a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:20.673 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.695+0000 7f2fb984a640 1 -- 172.21.15.114:0/3258793205 shutdown_connections 2024-03-29T11:22:20.673 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.695+0000 7f2fb984a640 1 -- 172.21.15.114:0/3258793205 >> 172.21.15.114:0/3258793205 conn(0x7f2fb406d180 msgr2=0x7f2fb4066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:20.673 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.695+0000 7f2fb984a640 1 -- 172.21.15.114:0/3258793205 shutdown_connections 2024-03-29T11:22:20.673 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.695+0000 7f2fb984a640 1 -- 172.21.15.114:0/3258793205 wait complete. 2024-03-29T11:22:20.675 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.696+0000 7f2fb984a640 10 monclient: build_initial_monmap 2024-03-29T11:22:20.675 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.696+0000 7f2fb984a640 10 monclient: monmap: 2024-03-29T11:22:20.675 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:22:20.675 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:20.675 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T11:22:20.675 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T11:22:20.675 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:20.675 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:22:20.675 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:22:20.675 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T11:22:20.675 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T11:22:20.675 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:22:20.675 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.697+0000 7f2fb984a640 1 librados: starting msgr at 2024-03-29T11:22:20.675 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.697+0000 7f2fb984a640 1 librados: starting objecter 2024-03-29T11:22:20.675 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.697+0000 7f2fb984a640 1 Processor -- start 2024-03-29T11:22:20.676 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.697+0000 7f2fb984a640 1 -- start start 2024-03-29T11:22:20.676 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.697+0000 7f2fb984a640 1 librados: setting wanted keys 2024-03-29T11:22:20.676 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.698+0000 7f2fb984a640 1 librados: calling monclient init 2024-03-29T11:22:20.676 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.698+0000 7f2fb984a640 10 monclient: init 2024-03-29T11:22:20.676 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.698+0000 7f2fb984a640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:20.676 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.699+0000 7f2fb984a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:22:20.678 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.699+0000 7f2fb984a640 10 monclient(hunting): picked mon.noname-b con 0x7f2fb40b2620 addr v1:172.21.15.176:6789/0 2024-03-29T11:22:20.678 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.699+0000 7f2fb984a640 10 monclient(hunting): picked mon.noname-a con 0x7f2fb4149b50 addr v1:172.21.15.114:6789/0 2024-03-29T11:22:20.679 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.699+0000 7f2fb984a640 10 monclient(hunting): picked mon.noname-c con 0x7f2fb414fc70 addr v1:172.21.15.114:6790/0 2024-03-29T11:22:20.679 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.700+0000 7f2fb984a640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fb40b7040 con 0x7f2fb4149b50 2024-03-29T11:22:20.679 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.700+0000 7f2fb984a640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fb40b74d0 con 0x7f2fb40b2620 2024-03-29T11:22:20.679 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.700+0000 7f2fb984a640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fb413e490 con 0x7f2fb414fc70 2024-03-29T11:22:20.679 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.700+0000 7f2fb984a640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:20.679 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.700+0000 7f2fb984a640 10 monclient(hunting): authenticate will time out at 1010.882481s 2024-03-29T11:22:20.679 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.700+0000 7f2fb2ffd640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f2fb40b2620 0x7f2fb40b6160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.114:44422/0 (socket says 172.21.15.114:44422) 2024-03-29T11:22:20.679 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.700+0000 7f2fb2ffd640 1 -- 172.21.15.114:0/685801956 learned_addr learned my addr 172.21.15.114:0/685801956 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:22:20.679 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.700+0000 7f2fb0ff9640 10 client.?.objecter ms_handle_connect 0x7f2fb40b2620 2024-03-29T11:22:20.679 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.700+0000 7f2fb0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:20.679 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.701+0000 7f2fb0ff9640 1 -- 172.21.15.114:0/685801956 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1777594977 0 0) 0x7f2fb40b74d0 con 0x7f2fb40b2620 2024-03-29T11:22:20.679 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.701+0000 7f2fb0ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:22:20.680 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.701+0000 7f2fb0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:20.680 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.701+0000 7f2fb0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:20.680 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.701+0000 7f2fb0ff9640 10 monclient(hunting): my global_id is 4276 2024-03-29T11:22:20.680 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.701+0000 7f2fb0ff9640 1 -- 172.21.15.114:0/685801956 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f94004eb0 con 0x7f2fb40b2620 2024-03-29T11:22:20.680 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.702+0000 7f2fb0ff9640 1 -- 172.21.15.114:0/685801956 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3682316396 0 0) 0x7f2f94004eb0 con 0x7f2fb40b2620 2024-03-29T11:22:20.680 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.702+0000 7f2fb0ff9640 1 -- 172.21.15.114:0/685801956 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2fb40b74d0 con 0x7f2fb40b2620 2024-03-29T11:22:20.680 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.702+0000 7f2fb0ff9640 1 -- 172.21.15.114:0/685801956 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f2fa4001960 con 0x7f2fb40b2620 2024-03-29T11:22:20.680 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.702+0000 7f2fb0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:20.680 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.702+0000 7f2fb0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:22:20.680 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.702+0000 7f2fb0ff9640 10 monclient(hunting): dump: 2024-03-29T11:22:20.680 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:22:20.680 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:22:20.680 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:22:20.680 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:22:20.681 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:20.681 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:22:20.681 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:22:20.681 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:22:20.681 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:22:20.681 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:22:20.681 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.702+0000 7f2fb0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:20.685 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.703+0000 7f2fb0ff9640 10 client.?.objecter ms_handle_connect 0x7f2fb4149b50 2024-03-29T11:22:20.685 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.703+0000 7f2fb0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:20.685 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.703+0000 7f2fb0ff9640 1 -- 172.21.15.114:0/685801956 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 324933444 0 0) 0x7f2fb40b7040 con 0x7f2fb4149b50 2024-03-29T11:22:20.685 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.703+0000 7f2fb0ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:22:20.685 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.703+0000 7f2fb0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:20.685 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.703+0000 7f2fb0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:20.685 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.703+0000 7f2fb0ff9640 10 monclient(hunting): my global_id is 4284 2024-03-29T11:22:20.685 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.703+0000 7f2fb0ff9640 1 -- 172.21.15.114:0/685801956 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f94004eb0 con 0x7f2fb4149b50 2024-03-29T11:22:20.685 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.704+0000 7f2fb0ff9640 1 -- 172.21.15.114:0/685801956 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2931355303 0 0) 0x7f2f94004eb0 con 0x7f2fb4149b50 2024-03-29T11:22:20.685 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.704+0000 7f2fb0ff9640 1 -- 172.21.15.114:0/685801956 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2fb40b7040 con 0x7f2fb4149b50 2024-03-29T11:22:20.686 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.704+0000 7f2fb0ff9640 1 -- 172.21.15.114:0/685801956 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f2f9c0014b0 con 0x7f2fb4149b50 2024-03-29T11:22:20.686 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.704+0000 7f2fb0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:20.686 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.704+0000 7f2fb0ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:22:20.686 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.704+0000 7f2fb0ff9640 10 monclient(hunting): dump: 2024-03-29T11:22:20.686 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:22:20.686 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:22:20.686 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:22:20.686 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:22:20.686 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:20.686 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:22:20.686 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:22:20.686 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:22:20.686 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:22:20.686 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:22:20.686 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.704+0000 7f2fb0ff9640 1 -- 172.21.15.114:0/685801956 <== mon.0 v1:172.21.15.114:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 506675350 0 0) 0x7f2fb40b7040 con 0x7f2fb4149b50 2024-03-29T11:22:20.687 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.704+0000 7f2fb0ff9640 1 -- 172.21.15.114:0/685801956 >> v1:172.21.15.114:6790/0 conn(0x7f2fb414fc70 legacy=0x7f2fb41e6630 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:22:20.687 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.705+0000 7f2fb0ff9640 1 -- 172.21.15.114:0/685801956 >> v1:172.21.15.176:6789/0 conn(0x7f2fb40b2620 legacy=0x7f2fb40b6160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:20.687 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.705+0000 7f2fb0ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:22:20.687 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.705+0000 7f2fb0ff9640 1 monclient: found mon.a 2024-03-29T11:22:20.687 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.705+0000 7f2fb0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:20.687 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.705+0000 7f2fb0ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T11:22:20.687 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.705+0000 7f2fb0ff9640 1 -- 172.21.15.114:0/685801956 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fb40b6920 con 0x7f2fb4149b50 2024-03-29T11:22:20.687 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.705+0000 7f2fb0ff9640 10 monclient: _finish_auth 0 2024-03-29T11:22:20.687 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.705+0000 7f2fb0ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:22:20.687 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.705+0000 7f2fb0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:20.687 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.705+0000 7f2fb0ff9640 1 -- 172.21.15.114:0/685801956 <== mon.0 v1:172.21.15.114:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2f9c004980 con 0x7f2fb4149b50 2024-03-29T11:22:20.687 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.705+0000 7f2fb0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:20.687 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.705+0000 7f2fb0ff9640 1 -- 172.21.15.114:0/685801956 <== mon.0 v1:172.21.15.114:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f2f9c004c90 con 0x7f2fb4149b50 2024-03-29T11:22:20.687 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.705+0000 7f2fb0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:20.687 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.705+0000 7f2fb1ffb640 4 set_mon_vals no callback set 2024-03-29T11:22:20.687 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.705+0000 7f2fb0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:22:20.687 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.705+0000 7f2fb0ff9640 10 monclient: dump: 2024-03-29T11:22:20.688 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:22:20.688 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:22:20.688 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:22:20.688 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:22:20.688 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:20.688 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:22:20.688 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:22:20.688 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:22:20.688 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:22:20.688 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:22:20.688 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.705+0000 7f2fb984a640 5 monclient: authenticate success, global_id 4284 2024-03-29T11:22:20.688 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.705+0000 7f2fb984a640 10 monclient: _renew_subs 2024-03-29T11:22:20.688 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.705+0000 7f2fb984a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T11:22:20.688 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.705+0000 7f2fb984a640 1 -- 172.21.15.114:0/685801956 --> v1:172.21.15.114:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fb40b6b10 con 0x7f2fb4149b50 2024-03-29T11:22:20.689 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.705+0000 7f2fb984a640 10 client.4284.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:20.689 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.705+0000 7f2fb984a640 10 monclient: _renew_subs 2024-03-29T11:22:20.689 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.705+0000 7f2fb984a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T11:22:20.690 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.705+0000 7f2fb984a640 1 -- 172.21.15.114:0/685801956 --> v1:172.21.15.114:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fb406fe20 con 0x7f2fb4149b50 2024-03-29T11:22:20.690 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.705+0000 7f2fb984a640 1 librados: init done 2024-03-29T11:22:20.690 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.707+0000 7f2fa9ffb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:20.690 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.707+0000 7f2fa9ffb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:20.690 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.707+0000 7f2fa9ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T11:22:20.690 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.707+0000 7f2fa9ffb640 1 -- 172.21.15.114:0/685801956 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f8c005080 con 0x7f2fb4149b50 2024-03-29T11:22:20.690 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.708+0000 7f2fb0ff9640 1 -- 172.21.15.114:0/685801956 <== mon.0 v1:172.21.15.114:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1594292461 0 0) 0x7f2f9c01ac60 con 0x7f2fb4149b50 2024-03-29T11:22:20.690 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.708+0000 7f2fb0ff9640 1 -- 172.21.15.114:0/685801956 <== mon.0 v1:172.21.15.114:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3238987313 0 0) 0x7f2f9c0849b0 con 0x7f2fb4149b50 2024-03-29T11:22:20.690 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.708+0000 7f2fb0ff9640 10 client.4284.objecter ms_dispatch 0x7f2fb4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:22:20.690 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.708+0000 7f2fb0ff9640 3 client.4284.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:22:20.690 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.708+0000 7f2fb0ff9640 3 client.4284.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:22:20.690 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.708+0000 7f2fb0ff9640 20 client.4284.objecter dump_active .. 0 homeless 2024-03-29T11:22:20.690 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.711+0000 7f2fb2ffd640 10 monclient: get_auth_request con 0x7f2f9406de60 auth_method 0 2024-03-29T11:22:20.690 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.712+0000 7f2fb0ff9640 10 client.4284.objecter ms_handle_connect 0x7f2f9406de60 2024-03-29T11:22:20.698 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.721+0000 7f2fb0ff9640 1 -- 172.21.15.114:0/685801956 <== mon.0 v1:172.21.15.114:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2f9c0503d0 con 0x7f2fb4149b50 2024-03-29T11:22:20.698 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.721+0000 7f2fb0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:20.698 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:20.721+0000 7f2fb0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:21.024 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:21.046+0000 7f2fa9ffb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:21.024 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:21.046+0000 7f2fa9ffb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:21.024 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:21.046+0000 7f2fa9ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T11:22:21.024 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:21.046+0000 7f2fa9ffb640 1 -- 172.21.15.114:0/685801956 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2f8c005330 con 0x7f2fb4149b50 2024-03-29T11:22:21.025 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:21.048+0000 7f2fb0ff9640 1 -- 172.21.15.114:0/685801956 <== mon.0 v1:172.21.15.114:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2858224613) 0x7f2f9c052300 con 0x7f2fb4149b50 2024-03-29T11:22:21.025 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:21.048+0000 7f2fb0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:21.025 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:21.048+0000 7f2fb0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:21.025 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T11:22:21.025 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"b0de2b9b-c45a-44c7-b6ee-e83b39a1e108","created":"2024-03-29T11:20:49.420660+0000","modified":"2024-03-29T11:20:51.700167+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:51.700167+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a64785d3-6b11-499a-b706-cc827ab96a74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5882dadb-e7a8-4604-ba9a-418013c106c8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d7e8c409-bc66-4e4a-bbdb-420bad31388e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0682989-3595-4cd6-90a5-ab9111c278e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"67068120-e3ff-4817-bd06-44feb7479a35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09e8b-8a85-4997-bdea-75d754e6ee67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2f0d9ead-f47f-4cb3-92a4-0a5ad23cd831","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fdb6a25f-6bd8-4c65-9b9b-b460ba7673a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:21.028 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:21.052+0000 7f2fa9ffb640 10 librados: watch_flush enter 2024-03-29T11:22:21.028 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:21.052+0000 7f2fa9ffb640 10 librados: watch_flush exit 2024-03-29T11:22:21.028 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:21.052+0000 7f2fa9ffb640 20 client.4284.objecter shutdown clearing up homeless session... 2024-03-29T11:22:21.028 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:21.052+0000 7f2fa9ffb640 10 client.4284.objecter successfully canceled tick 2024-03-29T11:22:21.029 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:21.053+0000 7f2fa9ffb640 1 -- 172.21.15.114:0/685801956 >> v1:172.21.15.114:6816/4038755055 conn(0x7f2f9406de60 legacy=0x7f2f94070320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:21.029 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:21.053+0000 7f2fa9ffb640 10 monclient: shutdown 2024-03-29T11:22:21.030 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:21.053+0000 7f2fa9ffb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:21.030 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:21.053+0000 7f2fa9ffb640 1 -- 172.21.15.114:0/685801956 >> v1:172.21.15.114:6789/0 conn(0x7f2fb4149b50 legacy=0x7f2fb406f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:21.031 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:21.053+0000 7f2fa9ffb640 1 -- 172.21.15.114:0/685801956 shutdown_connections 2024-03-29T11:22:21.031 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:21.053+0000 7f2fa9ffb640 1 -- 172.21.15.114:0/685801956 >> 172.21.15.114:0/685801956 conn(0x7f2fb406d180 msgr2=0x7f2fb40b43d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:21.031 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:21.054+0000 7f2fa9ffb640 1 -- 172.21.15.114:0/685801956 shutdown_connections 2024-03-29T11:22:21.031 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:21.054+0000 7f2fa9ffb640 1 -- 172.21.15.114:0/685801956 wait complete. 2024-03-29T11:22:21.032 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:21.055+0000 7f2fa9ffb640 1 librados: shutdown 2024-03-29T11:22:21.051 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:27.859 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:28.011 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.034+0000 7f9c59ca8640 10 monclient: get_monmap_and_config 2024-03-29T11:22:28.011 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.034+0000 7f9c59ca8640 10 monclient: build_initial_monmap 2024-03-29T11:22:28.011 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.034+0000 7f9c59ca8640 10 monclient: monmap: 2024-03-29T11:22:28.011 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:22:28.011 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:22:28.011 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:22:28.035593+0000 2024-03-29T11:22:28.011 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:22:28.035593+0000 2024-03-29T11:22:28.011 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:28.011 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:22:28.011 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:22:28.011 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T11:22:28.011 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T11:22:28.012 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:22:28.012 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.036+0000 7f9c59ca8640 1 Processor -- start 2024-03-29T11:22:28.012 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.036+0000 7f9c59ca8640 1 -- start start 2024-03-29T11:22:28.012 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.036+0000 7f9c59ca8640 10 monclient: init 2024-03-29T11:22:28.013 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.037+0000 7f9c59ca8640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:28.013 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.037+0000 7f9c59ca8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:22:28.018 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.040+0000 7f9c537fe640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f9c541457f0 0x7f9c5413ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:60773/0 (socket says 172.21.15.114:60773) 2024-03-29T11:22:28.018 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.040+0000 7f9c537fe640 1 -- 172.21.15.114:0/145771197 learned_addr learned my addr 172.21.15.114:0/145771197 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:22:28.018 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.040+0000 7f9c59ca8640 10 monclient(hunting): picked mon.noname-a con 0x7f9c541457f0 addr v1:172.21.15.114:6789/0 2024-03-29T11:22:28.018 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.040+0000 7f9c59ca8640 10 monclient(hunting): picked mon.noname-b con 0x7f9c5413f650 addr v1:172.21.15.114:6790/0 2024-03-29T11:22:28.020 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.043+0000 7f9c59ca8640 10 monclient(hunting): picked mon.noname-c con 0x7f9c540a9d70 addr v1:172.21.15.176:6789/0 2024-03-29T11:22:28.020 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.043+0000 7f9c59ca8640 1 -- 172.21.15.114:0/145771197 --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c540aaef0 con 0x7f9c541457f0 2024-03-29T11:22:28.020 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.043+0000 7f9c59ca8640 1 -- 172.21.15.114:0/145771197 --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c54145bd0 con 0x7f9c540a9d70 2024-03-29T11:22:28.020 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.043+0000 7f9c59ca8640 1 -- 172.21.15.114:0/145771197 --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c54145eb0 con 0x7f9c5413f650 2024-03-29T11:22:28.020 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.043+0000 7f9c59ca8640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:28.020 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.043+0000 7f9c59ca8640 10 monclient(hunting): authenticate will time out at 1018.226005s 2024-03-29T11:22:28.021 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.044+0000 7f9c527fc640 1 -- 172.21.15.114:0/145771197 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4067516902 0 0) 0x7f9c540aaef0 con 0x7f9c541457f0 2024-03-29T11:22:28.021 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.045+0000 7f9c527fc640 10 monclient(hunting): _negotiate 2024-03-29T11:22:28.021 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.045+0000 7f9c527fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:28.021 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.045+0000 7f9c527fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:28.021 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.045+0000 7f9c527fc640 10 monclient(hunting): my global_id is 4287 2024-03-29T11:22:28.021 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.045+0000 7f9c527fc640 1 -- 172.21.15.114:0/145771197 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c3c003520 con 0x7f9c541457f0 2024-03-29T11:22:28.022 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.046+0000 7f9c527fc640 1 -- 172.21.15.114:0/145771197 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3946743266 0 0) 0x7f9c3c003520 con 0x7f9c541457f0 2024-03-29T11:22:28.022 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.046+0000 7f9c527fc640 1 -- 172.21.15.114:0/145771197 >> v1:172.21.15.114:6790/0 conn(0x7f9c5413f650 legacy=0x7f9c540b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:22:28.022 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.046+0000 7f9c527fc640 1 -- 172.21.15.114:0/145771197 >> v1:172.21.15.176:6789/0 conn(0x7f9c540a9d70 legacy=0x7f9c5414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:28.022 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.046+0000 7f9c527fc640 10 monclient: _finish_hunting 0 2024-03-29T11:22:28.022 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.046+0000 7f9c527fc640 1 monclient: found mon.noname-a 2024-03-29T11:22:28.023 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.046+0000 7f9c527fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:28.023 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.046+0000 7f9c527fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.114:6789/0 2024-03-29T11:22:28.023 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.046+0000 7f9c527fc640 1 -- 172.21.15.114:0/145771197 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c54146340 con 0x7f9c541457f0 2024-03-29T11:22:28.023 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.046+0000 7f9c527fc640 10 monclient: _finish_auth 0 2024-03-29T11:22:28.023 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.046+0000 7f9c527fc640 10 monclient: _check_auth_tickets 2024-03-29T11:22:28.023 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.046+0000 7f9c527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:28.023 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.046+0000 7f9c527fc640 1 -- 172.21.15.114:0/145771197 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f9c4c001a10 con 0x7f9c541457f0 2024-03-29T11:22:28.023 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.046+0000 7f9c527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:28.023 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.046+0000 7f9c527fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:22:28.023 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.046+0000 7f9c527fc640 10 monclient: dump: 2024-03-29T11:22:28.023 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:22:28.023 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:22:28.023 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:22:28.023 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:22:28.023 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:28.024 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:22:28.024 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:22:28.024 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:22:28.024 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:22:28.024 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:22:28.024 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.046+0000 7f9c527fc640 1 -- 172.21.15.114:0/145771197 <== mon.0 v1:172.21.15.114:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9c4c001be0 con 0x7f9c541457f0 2024-03-29T11:22:28.024 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.046+0000 7f9c527fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:28.024 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.046+0000 7f9c527fc640 1 -- 172.21.15.114:0/145771197 <== mon.0 v1:172.21.15.114:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f9c4c002030 con 0x7f9c541457f0 2024-03-29T11:22:28.024 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.046+0000 7f9c527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:28.024 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.046+0000 7f9c527fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:22:28.024 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.046+0000 7f9c527fc640 10 monclient: dump: 2024-03-29T11:22:28.025 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:22:28.025 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:22:28.025 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:22:28.025 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:22:28.025 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:28.025 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:22:28.025 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:22:28.025 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:22:28.025 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:22:28.025 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:22:28.026 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.049+0000 7f9c59ca8640 5 monclient: authenticate success, global_id 4287 2024-03-29T11:22:28.026 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.049+0000 7f9c59ca8640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:28.026 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.049+0000 7f9c59ca8640 4 set_mon_vals no callback set 2024-03-29T11:22:28.026 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.050+0000 7f9c59ca8640 10 monclient: shutdown 2024-03-29T11:22:28.026 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.050+0000 7f9c59ca8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:28.026 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.050+0000 7f9c59ca8640 1 -- 172.21.15.114:0/145771197 >> v1:172.21.15.114:6789/0 conn(0x7f9c541457f0 legacy=0x7f9c5413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:28.028 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.052+0000 7f9c59ca8640 1 -- 172.21.15.114:0/145771197 shutdown_connections 2024-03-29T11:22:28.028 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.052+0000 7f9c59ca8640 1 -- 172.21.15.114:0/145771197 >> 172.21.15.114:0/145771197 conn(0x7f9c54066a00 msgr2=0x7f9c54066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:28.031 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.054+0000 7f9c59ca8640 1 -- 172.21.15.114:0/145771197 shutdown_connections 2024-03-29T11:22:28.034 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.055+0000 7f9c59ca8640 1 -- 172.21.15.114:0/145771197 wait complete. 2024-03-29T11:22:28.034 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.056+0000 7f9c59ca8640 10 monclient: build_initial_monmap 2024-03-29T11:22:28.034 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.056+0000 7f9c59ca8640 10 monclient: monmap: 2024-03-29T11:22:28.034 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:22:28.034 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:28.034 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T11:22:28.034 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T11:22:28.034 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:28.034 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:22:28.034 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:22:28.034 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T11:22:28.035 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T11:22:28.035 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:22:28.035 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.057+0000 7f9c59ca8640 1 librados: starting msgr at 2024-03-29T11:22:28.035 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.057+0000 7f9c59ca8640 1 librados: starting objecter 2024-03-29T11:22:28.035 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.057+0000 7f9c59ca8640 1 Processor -- start 2024-03-29T11:22:28.035 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.057+0000 7f9c59ca8640 1 -- start start 2024-03-29T11:22:28.035 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.057+0000 7f9c59ca8640 1 librados: setting wanted keys 2024-03-29T11:22:28.035 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.057+0000 7f9c59ca8640 1 librados: calling monclient init 2024-03-29T11:22:28.035 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.057+0000 7f9c59ca8640 10 monclient: init 2024-03-29T11:22:28.036 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.058+0000 7f9c59ca8640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:28.036 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.058+0000 7f9c59ca8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:22:28.037 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.061+0000 7f9c59ca8640 10 monclient(hunting): picked mon.noname-b con 0x7f9c540a9d70 addr v1:172.21.15.176:6789/0 2024-03-29T11:22:28.039 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.063+0000 7f9c59ca8640 10 monclient(hunting): picked mon.noname-a con 0x7f9c5413f650 addr v1:172.21.15.114:6789/0 2024-03-29T11:22:28.039 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.063+0000 7f9c59ca8640 10 monclient(hunting): picked mon.noname-c con 0x7f9c541457f0 addr v1:172.21.15.114:6790/0 2024-03-29T11:22:28.039 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.063+0000 7f9c59ca8640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c54139520 con 0x7f9c5413f650 2024-03-29T11:22:28.039 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.063+0000 7f9c59ca8640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c54070e20 con 0x7f9c540a9d70 2024-03-29T11:22:28.040 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.063+0000 7f9c59ca8640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c54071100 con 0x7f9c541457f0 2024-03-29T11:22:28.040 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.063+0000 7f9c59ca8640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:28.040 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.063+0000 7f9c59ca8640 10 monclient(hunting): authenticate will time out at 1018.245351s 2024-03-29T11:22:28.040 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.063+0000 7f9c537fe640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f9c540a9d70 0x7f9c54071890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.114:45549/0 (socket says 172.21.15.114:45549) 2024-03-29T11:22:28.040 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.063+0000 7f9c537fe640 1 -- 172.21.15.114:0/3264895282 learned_addr learned my addr 172.21.15.114:0/3264895282 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:22:28.041 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.065+0000 7f9c3bfff640 10 client.?.objecter ms_handle_connect 0x7f9c540a9d70 2024-03-29T11:22:28.041 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.065+0000 7f9c3bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:28.042 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.066+0000 7f9c3bfff640 10 client.?.objecter ms_handle_connect 0x7f9c5413f650 2024-03-29T11:22:28.042 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.066+0000 7f9c3bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:28.042 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.066+0000 7f9c3bfff640 1 -- 172.21.15.114:0/3264895282 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 143987514 0 0) 0x7f9c54070e20 con 0x7f9c540a9d70 2024-03-29T11:22:28.043 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.066+0000 7f9c3bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:22:28.043 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.066+0000 7f9c3bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:28.043 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.066+0000 7f9c3bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:28.043 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.066+0000 7f9c3bfff640 10 monclient(hunting): my global_id is 4282 2024-03-29T11:22:28.043 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.066+0000 7f9c3bfff640 1 -- 172.21.15.114:0/3264895282 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c40004eb0 con 0x7f9c540a9d70 2024-03-29T11:22:28.048 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.066+0000 7f9c3bfff640 10 client.?.objecter ms_handle_connect 0x7f9c541457f0 2024-03-29T11:22:28.048 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.067+0000 7f9c3bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:28.048 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.067+0000 7f9c3bfff640 1 -- 172.21.15.114:0/3264895282 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1984717029 0 0) 0x7f9c40004eb0 con 0x7f9c540a9d70 2024-03-29T11:22:28.048 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.067+0000 7f9c3bfff640 1 -- 172.21.15.114:0/3264895282 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9c54070e20 con 0x7f9c540a9d70 2024-03-29T11:22:28.048 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.067+0000 7f9c3bfff640 1 -- 172.21.15.114:0/3264895282 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f9c4c001fb0 con 0x7f9c540a9d70 2024-03-29T11:22:28.048 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.067+0000 7f9c3bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:28.048 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.067+0000 7f9c3bfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:22:28.048 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.067+0000 7f9c3bfff640 10 monclient(hunting): dump: 2024-03-29T11:22:28.048 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:22:28.048 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:22:28.048 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:22:28.048 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:22:28.049 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:28.049 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:22:28.049 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:22:28.049 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:22:28.049 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:22:28.049 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:22:28.049 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.067+0000 7f9c3bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:28.049 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.067+0000 7f9c3bfff640 1 -- 172.21.15.114:0/3264895282 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2737443873 0 0) 0x7f9c54071100 con 0x7f9c541457f0 2024-03-29T11:22:28.049 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.067+0000 7f9c3bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:22:28.049 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.067+0000 7f9c3bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:28.049 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.067+0000 7f9c3bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:28.049 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.067+0000 7f9c3bfff640 10 monclient(hunting): my global_id is 4277 2024-03-29T11:22:28.049 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.067+0000 7f9c3bfff640 1 -- 172.21.15.114:0/3264895282 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c40004eb0 con 0x7f9c541457f0 2024-03-29T11:22:28.049 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.067+0000 7f9c3bfff640 1 -- 172.21.15.114:0/3264895282 <== mon.1 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1568968190 0 0) 0x7f9c54070e20 con 0x7f9c540a9d70 2024-03-29T11:22:28.050 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.067+0000 7f9c3bfff640 1 -- 172.21.15.114:0/3264895282 >> v1:172.21.15.114:6790/0 conn(0x7f9c541457f0 legacy=0x7f9c5406f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:28.050 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.067+0000 7f9c3bfff640 1 -- 172.21.15.114:0/3264895282 >> v1:172.21.15.114:6789/0 conn(0x7f9c5413f650 legacy=0x7f9c5406ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:28.050 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.067+0000 7f9c3bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:22:28.050 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.067+0000 7f9c3bfff640 1 monclient: found mon.b 2024-03-29T11:22:28.050 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.067+0000 7f9c3bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:28.050 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.067+0000 7f9c3bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:22:28.050 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.067+0000 7f9c3bfff640 1 -- 172.21.15.114:0/3264895282 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c541391b0 con 0x7f9c540a9d70 2024-03-29T11:22:28.050 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.067+0000 7f9c3bfff640 10 monclient: _finish_auth 0 2024-03-29T11:22:28.050 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.068+0000 7f9c3bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:22:28.050 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.068+0000 7f9c3bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:28.051 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.068+0000 7f9c59ca8640 5 monclient: authenticate success, global_id 4282 2024-03-29T11:22:28.051 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.068+0000 7f9c59ca8640 10 monclient: _renew_subs 2024-03-29T11:22:28.051 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.068+0000 7f9c59ca8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:22:28.051 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.068+0000 7f9c59ca8640 1 -- 172.21.15.114:0/3264895282 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c54071fa0 con 0x7f9c540a9d70 2024-03-29T11:22:28.051 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.068+0000 7f9c59ca8640 10 client.4282.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:28.051 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.068+0000 7f9c59ca8640 10 monclient: _renew_subs 2024-03-29T11:22:28.051 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.068+0000 7f9c59ca8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:22:28.051 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.068+0000 7f9c59ca8640 1 -- 172.21.15.114:0/3264895282 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c541e7210 con 0x7f9c540a9d70 2024-03-29T11:22:28.051 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.068+0000 7f9c59ca8640 1 librados: init done 2024-03-29T11:22:28.052 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.068+0000 7f9c3bfff640 1 -- 172.21.15.114:0/3264895282 <== mon.1 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9c4c004e10 con 0x7f9c540a9d70 2024-03-29T11:22:28.052 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.068+0000 7f9c3bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:28.052 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.068+0000 7f9c3bfff640 1 -- 172.21.15.114:0/3264895282 <== mon.1 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f9c4c005160 con 0x7f9c540a9d70 2024-03-29T11:22:28.052 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.068+0000 7f9c517fa640 4 set_mon_vals no callback set 2024-03-29T11:22:28.052 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.069+0000 7f9c3bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:28.052 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.069+0000 7f9c3bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:22:28.052 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.069+0000 7f9c3bfff640 10 monclient: dump: 2024-03-29T11:22:28.052 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:22:28.052 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:22:28.053 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:22:28.053 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:22:28.053 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:28.053 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:22:28.053 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:22:28.053 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:22:28.053 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:22:28.053 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:22:28.053 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.071+0000 7f9c59ca8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:28.053 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.071+0000 7f9c59ca8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:28.053 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.071+0000 7f9c59ca8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:22:28.053 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.071+0000 7f9c59ca8640 1 -- 172.21.15.114:0/3264895282 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c34005080 con 0x7f9c540a9d70 2024-03-29T11:22:28.053 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.073+0000 7f9c3bfff640 1 -- 172.21.15.114:0/3264895282 <== mon.1 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1594292461 0 0) 0x7f9c4c01b130 con 0x7f9c540a9d70 2024-03-29T11:22:28.053 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.074+0000 7f9c3bfff640 1 -- 172.21.15.114:0/3264895282 <== mon.1 v1:172.21.15.176:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3238987313 0 0) 0x7f9c4c084ed0 con 0x7f9c540a9d70 2024-03-29T11:22:28.053 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.074+0000 7f9c3bfff640 10 client.4282.objecter ms_dispatch 0x7f9c54000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:22:28.054 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.074+0000 7f9c3bfff640 3 client.4282.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:22:28.054 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.074+0000 7f9c3bfff640 3 client.4282.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:22:28.054 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.074+0000 7f9c3bfff640 20 client.4282.objecter dump_active .. 0 homeless 2024-03-29T11:22:28.054 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.074+0000 7f9c52ffd640 10 monclient: get_auth_request con 0x7f9c4006ddf0 auth_method 0 2024-03-29T11:22:28.054 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.075+0000 7f9c3bfff640 10 client.4282.objecter ms_handle_connect 0x7f9c4006ddf0 2024-03-29T11:22:28.056 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.080+0000 7f9c3bfff640 1 -- 172.21.15.114:0/3264895282 <== mon.1 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9c4c0507b0 con 0x7f9c540a9d70 2024-03-29T11:22:28.056 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.080+0000 7f9c3bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:28.056 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.080+0000 7f9c3bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:28.375 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.399+0000 7f9c59ca8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:28.375 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.399+0000 7f9c59ca8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:28.375 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.399+0000 7f9c59ca8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:22:28.375 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.399+0000 7f9c59ca8640 1 -- 172.21.15.114:0/3264895282 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9c34005330 con 0x7f9c540a9d70 2024-03-29T11:22:28.379 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.403+0000 7f9c3bfff640 1 -- 172.21.15.114:0/3264895282 <== mon.1 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2858224613) 0x7f9c4c0526e0 con 0x7f9c540a9d70 2024-03-29T11:22:28.379 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.403+0000 7f9c3bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:28.380 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.403+0000 7f9c3bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:28.381 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T11:22:28.381 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"b0de2b9b-c45a-44c7-b6ee-e83b39a1e108","created":"2024-03-29T11:20:49.420660+0000","modified":"2024-03-29T11:20:51.700167+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:51.700167+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a64785d3-6b11-499a-b706-cc827ab96a74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5882dadb-e7a8-4604-ba9a-418013c106c8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d7e8c409-bc66-4e4a-bbdb-420bad31388e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0682989-3595-4cd6-90a5-ab9111c278e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"67068120-e3ff-4817-bd06-44feb7479a35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09e8b-8a85-4997-bdea-75d754e6ee67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2f0d9ead-f47f-4cb3-92a4-0a5ad23cd831","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fdb6a25f-6bd8-4c65-9b9b-b460ba7673a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:28.385 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.408+0000 7f9c59ca8640 10 librados: watch_flush enter 2024-03-29T11:22:28.385 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.408+0000 7f9c59ca8640 10 librados: watch_flush exit 2024-03-29T11:22:28.385 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.408+0000 7f9c59ca8640 20 client.4282.objecter shutdown clearing up homeless session... 2024-03-29T11:22:28.385 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.408+0000 7f9c59ca8640 10 client.4282.objecter successfully canceled tick 2024-03-29T11:22:28.385 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.408+0000 7f9c59ca8640 1 -- 172.21.15.114:0/3264895282 >> v1:172.21.15.114:6816/4038755055 conn(0x7f9c4006ddf0 legacy=0x7f9c400702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:28.386 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.409+0000 7f9c59ca8640 10 monclient: shutdown 2024-03-29T11:22:28.386 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.409+0000 7f9c59ca8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:28.386 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.409+0000 7f9c59ca8640 1 -- 172.21.15.114:0/3264895282 >> v1:172.21.15.176:6789/0 conn(0x7f9c540a9d70 legacy=0x7f9c54071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:28.386 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.409+0000 7f9c59ca8640 1 -- 172.21.15.114:0/3264895282 shutdown_connections 2024-03-29T11:22:28.386 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.409+0000 7f9c59ca8640 1 -- 172.21.15.114:0/3264895282 >> 172.21.15.114:0/3264895282 conn(0x7f9c54066a00 msgr2=0x7f9c5413c2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:28.386 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.409+0000 7f9c59ca8640 1 -- 172.21.15.114:0/3264895282 shutdown_connections 2024-03-29T11:22:28.387 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.409+0000 7f9c59ca8640 1 -- 172.21.15.114:0/3264895282 wait complete. 2024-03-29T11:22:28.387 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:28.410+0000 7f9c59ca8640 1 librados: shutdown 2024-03-29T11:22:28.403 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:35.209 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:35.362 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.385+0000 7f504ffff640 10 monclient: get_monmap_and_config 2024-03-29T11:22:35.362 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.385+0000 7f504ffff640 10 monclient: build_initial_monmap 2024-03-29T11:22:35.362 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.385+0000 7f504ffff640 10 monclient: monmap: 2024-03-29T11:22:35.362 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:22:35.362 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:22:35.362 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:22:35.386538+0000 2024-03-29T11:22:35.362 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:22:35.386538+0000 2024-03-29T11:22:35.362 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:35.362 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:22:35.363 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:22:35.363 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T11:22:35.363 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T11:22:35.363 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:22:35.364 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.386+0000 7f504ffff640 1 Processor -- start 2024-03-29T11:22:35.364 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.386+0000 7f504ffff640 1 -- start start 2024-03-29T11:22:35.364 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.386+0000 7f504ffff640 10 monclient: init 2024-03-29T11:22:35.364 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.387+0000 7f504ffff640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:35.364 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.387+0000 7f504ffff640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:22:35.364 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.387+0000 7f504ffff640 10 monclient(hunting): picked mon.noname-a con 0x7f5048007c80 addr v1:172.21.15.114:6789/0 2024-03-29T11:22:35.364 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.387+0000 7f504ffff640 10 monclient(hunting): picked mon.noname-c con 0x7f504802b4d0 addr v1:172.21.15.176:6789/0 2024-03-29T11:22:35.365 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.387+0000 7f504ffff640 10 monclient(hunting): picked mon.noname-b con 0x7f50480c1b30 addr v1:172.21.15.114:6790/0 2024-03-29T11:22:35.365 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.387+0000 7f504ffff640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f504802ad60 con 0x7f5048007c80 2024-03-29T11:22:35.365 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.387+0000 7f504ffff640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f504802b040 con 0x7f504802b4d0 2024-03-29T11:22:35.365 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.387+0000 7f504ffff640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f504802d220 con 0x7f50480c1b30 2024-03-29T11:22:35.365 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.387+0000 7f504ffff640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:35.365 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.387+0000 7f504ffff640 10 monclient(hunting): authenticate will time out at 1025.569363s 2024-03-29T11:22:35.369 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.392+0000 7f504e7fc640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f504802b4d0 0x7f50480c1420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.114:33693/0 (socket says 172.21.15.114:33693) 2024-03-29T11:22:35.369 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.392+0000 7f504e7fc640 1 -- 172.21.15.114:0/3378008343 learned_addr learned my addr 172.21.15.114:0/3378008343 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:22:35.369 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.393+0000 7f504dffb640 1 -- 172.21.15.114:0/3378008343 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2510070286 0 0) 0x7f504802b040 con 0x7f504802b4d0 2024-03-29T11:22:35.369 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.393+0000 7f504dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:22:35.369 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.393+0000 7f504dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:35.369 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.393+0000 7f504dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:35.369 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.393+0000 7f504dffb640 10 monclient(hunting): my global_id is 4285 2024-03-29T11:22:35.370 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.393+0000 7f504dffb640 1 -- 172.21.15.114:0/3378008343 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5038003520 con 0x7f504802b4d0 2024-03-29T11:22:35.370 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.393+0000 7f504dffb640 1 -- 172.21.15.114:0/3378008343 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1325820979 0 0) 0x7f504802ad60 con 0x7f5048007c80 2024-03-29T11:22:35.370 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.393+0000 7f504dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:22:35.370 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.393+0000 7f504dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:35.370 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.393+0000 7f504dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:35.371 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.393+0000 7f504dffb640 10 monclient(hunting): my global_id is 4293 2024-03-29T11:22:35.371 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.393+0000 7f504dffb640 1 -- 172.21.15.114:0/3378008343 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f504802b040 con 0x7f5048007c80 2024-03-29T11:22:35.371 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.393+0000 7f504dffb640 1 -- 172.21.15.114:0/3378008343 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3417549746 0 0) 0x7f5038003520 con 0x7f504802b4d0 2024-03-29T11:22:35.371 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.393+0000 7f504dffb640 1 -- 172.21.15.114:0/3378008343 >> v1:172.21.15.114:6790/0 conn(0x7f50480c1b30 legacy=0x7f50480cfda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:35.371 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.393+0000 7f504dffb640 1 -- 172.21.15.114:0/3378008343 >> v1:172.21.15.114:6789/0 conn(0x7f5048007c80 legacy=0x7f50480c7a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:35.371 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.393+0000 7f504dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:22:35.371 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.393+0000 7f504dffb640 1 monclient: found mon.noname-c 2024-03-29T11:22:35.371 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.393+0000 7f504dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:35.371 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.393+0000 7f504dffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.176:6789/0 2024-03-29T11:22:35.371 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.393+0000 7f504dffb640 1 -- 172.21.15.114:0/3378008343 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f504802d500 con 0x7f504802b4d0 2024-03-29T11:22:35.371 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.393+0000 7f504dffb640 10 monclient: _finish_auth 0 2024-03-29T11:22:35.371 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.393+0000 7f504dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:22:35.371 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.393+0000 7f504dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:35.371 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.393+0000 7f504dffb640 1 -- 172.21.15.114:0/3378008343 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f5044001690 con 0x7f504802b4d0 2024-03-29T11:22:35.372 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.393+0000 7f504dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:35.372 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.393+0000 7f504dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:22:35.372 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.393+0000 7f504dffb640 10 monclient: dump: 2024-03-29T11:22:35.372 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:22:35.372 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:22:35.372 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:22:35.372 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:22:35.372 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:35.372 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:22:35.372 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:22:35.372 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:22:35.372 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:22:35.372 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:22:35.372 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.393+0000 7f504ffff640 5 monclient: authenticate success, global_id 4285 2024-03-29T11:22:35.372 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.393+0000 7f504ffff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:35.375 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.397+0000 7f504dffb640 1 -- 172.21.15.114:0/3378008343 <== mon.1 v1:172.21.15.176:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f50440017f0 con 0x7f504802b4d0 2024-03-29T11:22:35.375 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.397+0000 7f504dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:35.375 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.397+0000 7f504dffb640 1 -- 172.21.15.114:0/3378008343 <== mon.1 v1:172.21.15.176:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f5044001b80 con 0x7f504802b4d0 2024-03-29T11:22:35.375 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.397+0000 7f504dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:35.375 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.397+0000 7f504dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:22:35.375 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.397+0000 7f504dffb640 10 monclient: dump: 2024-03-29T11:22:35.375 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:22:35.375 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:22:35.375 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:22:35.375 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:22:35.375 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:35.375 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:22:35.376 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:22:35.376 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:22:35.376 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:22:35.376 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:22:35.376 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.397+0000 7f504ffff640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:35.376 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.397+0000 7f504ffff640 4 set_mon_vals no callback set 2024-03-29T11:22:35.376 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.398+0000 7f504ffff640 10 monclient: shutdown 2024-03-29T11:22:35.376 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.398+0000 7f504ffff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:35.376 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.398+0000 7f504ffff640 1 -- 172.21.15.114:0/3378008343 >> v1:172.21.15.176:6789/0 conn(0x7f504802b4d0 legacy=0x7f50480c1420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:35.376 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.398+0000 7f504ffff640 1 -- 172.21.15.114:0/3378008343 shutdown_connections 2024-03-29T11:22:35.376 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.398+0000 7f504ffff640 1 -- 172.21.15.114:0/3378008343 >> 172.21.15.114:0/3378008343 conn(0x7f5048006a10 msgr2=0x7f5048006e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:35.376 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.398+0000 7f504ffff640 1 -- 172.21.15.114:0/3378008343 shutdown_connections 2024-03-29T11:22:35.376 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.398+0000 7f504ffff640 1 -- 172.21.15.114:0/3378008343 wait complete. 2024-03-29T11:22:35.377 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.399+0000 7f504ffff640 10 monclient: build_initial_monmap 2024-03-29T11:22:35.378 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.399+0000 7f504ffff640 10 monclient: monmap: 2024-03-29T11:22:35.378 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:22:35.378 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:35.378 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T11:22:35.378 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T11:22:35.378 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:35.378 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:22:35.378 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:22:35.378 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T11:22:35.378 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T11:22:35.378 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:22:35.378 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.400+0000 7f504ffff640 1 librados: starting msgr at 2024-03-29T11:22:35.378 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.400+0000 7f504ffff640 1 librados: starting objecter 2024-03-29T11:22:35.379 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.400+0000 7f504ffff640 1 Processor -- start 2024-03-29T11:22:35.379 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.400+0000 7f504ffff640 1 -- start start 2024-03-29T11:22:35.379 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.400+0000 7f504ffff640 1 librados: setting wanted keys 2024-03-29T11:22:35.379 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.400+0000 7f504ffff640 1 librados: calling monclient init 2024-03-29T11:22:35.379 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.400+0000 7f504ffff640 10 monclient: init 2024-03-29T11:22:35.379 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.401+0000 7f504ffff640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:35.379 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.401+0000 7f504ffff640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:22:35.379 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.401+0000 7f504ffff640 10 monclient(hunting): picked mon.noname-b con 0x7f5048007c80 addr v1:172.21.15.176:6789/0 2024-03-29T11:22:35.382 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.403+0000 7f504ffff640 10 monclient(hunting): picked mon.noname-a con 0x7f504802b4d0 addr v1:172.21.15.114:6789/0 2024-03-29T11:22:35.382 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.405+0000 7f504ffff640 10 monclient(hunting): picked mon.noname-c con 0x7f50480c1b30 addr v1:172.21.15.114:6790/0 2024-03-29T11:22:35.383 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.405+0000 7f504ffff640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50480c2c20 con 0x7f504802b4d0 2024-03-29T11:22:35.383 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.405+0000 7f504ffff640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50480c4690 con 0x7f5048007c80 2024-03-29T11:22:35.383 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.405+0000 7f504ffff640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50480c4820 con 0x7f50480c1b30 2024-03-29T11:22:35.383 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.405+0000 7f504ffff640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:35.383 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.405+0000 7f504ffff640 10 monclient(hunting): authenticate will time out at 1025.587876s 2024-03-29T11:22:35.383 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.405+0000 7f504effd640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f5048007c80 0x7f50480c2280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.114:33757/0 (socket says 172.21.15.114:33757) 2024-03-29T11:22:35.383 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.405+0000 7f504effd640 1 -- 172.21.15.114:0/120142017 learned_addr learned my addr 172.21.15.114:0/120142017 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:22:35.383 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.406+0000 7f50377fe640 10 client.?.objecter ms_handle_connect 0x7f5048007c80 2024-03-29T11:22:35.383 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.406+0000 7f50377fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:35.383 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.406+0000 7f50377fe640 10 client.?.objecter ms_handle_connect 0x7f504802b4d0 2024-03-29T11:22:35.383 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.406+0000 7f50377fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:35.385 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.408+0000 7f50377fe640 1 -- 172.21.15.114:0/120142017 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1083247147 0 0) 0x7f50480c2c20 con 0x7f504802b4d0 2024-03-29T11:22:35.386 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.408+0000 7f50377fe640 10 monclient(hunting): _negotiate 2024-03-29T11:22:35.386 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.408+0000 7f50377fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:35.386 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.408+0000 7f50377fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:35.386 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.408+0000 7f50377fe640 10 monclient(hunting): my global_id is 4296 2024-03-29T11:22:35.386 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.408+0000 7f50377fe640 1 -- 172.21.15.114:0/120142017 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f503c004eb0 con 0x7f504802b4d0 2024-03-29T11:22:35.386 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.408+0000 7f50377fe640 10 client.?.objecter ms_handle_connect 0x7f50480c1b30 2024-03-29T11:22:35.386 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.408+0000 7f50377fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:35.386 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.409+0000 7f50377fe640 1 -- 172.21.15.114:0/120142017 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4014407653 0 0) 0x7f50480c4690 con 0x7f5048007c80 2024-03-29T11:22:35.386 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.409+0000 7f50377fe640 10 monclient(hunting): _negotiate 2024-03-29T11:22:35.386 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.409+0000 7f50377fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:35.386 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.409+0000 7f50377fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:35.386 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.409+0000 7f50377fe640 10 monclient(hunting): my global_id is 4288 2024-03-29T11:22:35.386 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.409+0000 7f50377fe640 1 -- 172.21.15.114:0/120142017 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f503c0051d0 con 0x7f5048007c80 2024-03-29T11:22:35.387 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.409+0000 7f50377fe640 1 -- 172.21.15.114:0/120142017 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1805192196 0 0) 0x7f503c0051d0 con 0x7f5048007c80 2024-03-29T11:22:35.387 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.409+0000 7f50377fe640 1 -- 172.21.15.114:0/120142017 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f503c005640 con 0x7f5048007c80 2024-03-29T11:22:35.387 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.409+0000 7f50377fe640 1 -- 172.21.15.114:0/120142017 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f5040001a30 con 0x7f5048007c80 2024-03-29T11:22:35.387 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.409+0000 7f50377fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:35.387 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.409+0000 7f50377fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:22:35.387 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.409+0000 7f50377fe640 10 monclient(hunting): dump: 2024-03-29T11:22:35.387 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:22:35.387 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:22:35.387 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:22:35.387 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:22:35.388 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:35.388 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:22:35.388 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:22:35.388 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:22:35.388 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:22:35.388 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:22:35.388 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.409+0000 7f50377fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:35.390 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.412+0000 7f50377fe640 1 -- 172.21.15.114:0/120142017 <== mon.1 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4279803455 0 0) 0x7f503c005640 con 0x7f5048007c80 2024-03-29T11:22:35.390 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.412+0000 7f50377fe640 1 -- 172.21.15.114:0/120142017 >> v1:172.21.15.114:6790/0 conn(0x7f50480c1b30 legacy=0x7f50480bd7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:35.390 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.412+0000 7f50377fe640 1 -- 172.21.15.114:0/120142017 >> v1:172.21.15.114:6789/0 conn(0x7f504802b4d0 legacy=0x7f50480c3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:35.390 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.412+0000 7f50377fe640 10 monclient: _finish_hunting 0 2024-03-29T11:22:35.390 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.412+0000 7f50377fe640 1 monclient: found mon.b 2024-03-29T11:22:35.390 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.412+0000 7f50377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:35.390 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.412+0000 7f50377fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:22:35.391 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.412+0000 7f50377fe640 1 -- 172.21.15.114:0/120142017 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f504802da20 con 0x7f5048007c80 2024-03-29T11:22:35.391 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.412+0000 7f50377fe640 10 monclient: _finish_auth 0 2024-03-29T11:22:35.391 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.412+0000 7f50377fe640 10 monclient: _check_auth_tickets 2024-03-29T11:22:35.391 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.412+0000 7f50377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:35.391 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.412+0000 7f504ffff640 5 monclient: authenticate success, global_id 4288 2024-03-29T11:22:35.391 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.412+0000 7f504ffff640 10 monclient: _renew_subs 2024-03-29T11:22:35.391 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.412+0000 7f504ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:22:35.391 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.412+0000 7f504ffff640 1 -- 172.21.15.114:0/120142017 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f504802dc10 con 0x7f5048007c80 2024-03-29T11:22:35.391 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.412+0000 7f504ffff640 10 client.4288.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:35.391 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.412+0000 7f504ffff640 10 monclient: _renew_subs 2024-03-29T11:22:35.391 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.412+0000 7f504ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:22:35.391 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.412+0000 7f504ffff640 1 -- 172.21.15.114:0/120142017 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f50480c3320 con 0x7f5048007c80 2024-03-29T11:22:35.391 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.412+0000 7f504ffff640 1 librados: init done 2024-03-29T11:22:35.393 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.413+0000 7f50377fe640 1 -- 172.21.15.114:0/120142017 <== mon.1 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5040001c70 con 0x7f5048007c80 2024-03-29T11:22:35.393 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.413+0000 7f50377fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:35.393 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.413+0000 7f504d7fa640 4 set_mon_vals no callback set 2024-03-29T11:22:35.393 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.414+0000 7f50377fe640 1 -- 172.21.15.114:0/120142017 <== mon.1 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f5040003a60 con 0x7f5048007c80 2024-03-29T11:22:35.393 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.414+0000 7f50377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:35.393 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.414+0000 7f50377fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:22:35.394 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.414+0000 7f50377fe640 10 monclient: dump: 2024-03-29T11:22:35.394 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:22:35.394 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:22:35.394 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:22:35.394 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:22:35.394 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:35.394 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:22:35.394 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:22:35.394 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:22:35.394 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:22:35.394 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:22:35.394 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.414+0000 7f50357fa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:35.394 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.414+0000 7f50357fa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:35.394 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.414+0000 7f50357fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:22:35.395 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.414+0000 7f50357fa640 1 -- 172.21.15.114:0/120142017 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50480c5670 con 0x7f5048007c80 2024-03-29T11:22:35.395 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.415+0000 7f50377fe640 1 -- 172.21.15.114:0/120142017 <== mon.1 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1594292461 0 0) 0x7f5040003bc0 con 0x7f5048007c80 2024-03-29T11:22:35.395 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.416+0000 7f504e7fc640 10 monclient: get_auth_request con 0x7f503c06de20 auth_method 0 2024-03-29T11:22:35.395 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.417+0000 7f50377fe640 10 client.4288.objecter ms_handle_connect 0x7f503c06de20 2024-03-29T11:22:35.395 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.417+0000 7f50377fe640 1 -- 172.21.15.114:0/120142017 <== mon.1 v1:172.21.15.176:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3238987313 0 0) 0x7f5040054840 con 0x7f5048007c80 2024-03-29T11:22:35.396 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.417+0000 7f50377fe640 10 client.4288.objecter ms_dispatch 0x7f5050000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:22:35.396 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.417+0000 7f50377fe640 3 client.4288.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:22:35.396 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.417+0000 7f50377fe640 3 client.4288.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:22:35.396 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.417+0000 7f50377fe640 20 client.4288.objecter dump_active .. 0 homeless 2024-03-29T11:22:35.406 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.429+0000 7f50377fe640 1 -- 172.21.15.114:0/120142017 <== mon.1 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5040050ae0 con 0x7f5048007c80 2024-03-29T11:22:35.406 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.429+0000 7f50377fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:35.406 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.429+0000 7f50377fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:35.739 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.762+0000 7f50357fa640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:35.739 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.762+0000 7f50357fa640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:35.739 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.762+0000 7f50357fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:22:35.739 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.762+0000 7f50357fa640 1 -- 172.21.15.114:0/120142017 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f50480008d0 con 0x7f5048007c80 2024-03-29T11:22:35.741 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.765+0000 7f50377fe640 1 -- 172.21.15.114:0/120142017 <== mon.1 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2858224613) 0x7f5040050c80 con 0x7f5048007c80 2024-03-29T11:22:35.741 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.765+0000 7f50377fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:35.741 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.765+0000 7f50377fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:35.744 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T11:22:35.744 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"b0de2b9b-c45a-44c7-b6ee-e83b39a1e108","created":"2024-03-29T11:20:49.420660+0000","modified":"2024-03-29T11:20:51.700167+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:51.700167+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a64785d3-6b11-499a-b706-cc827ab96a74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5882dadb-e7a8-4604-ba9a-418013c106c8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d7e8c409-bc66-4e4a-bbdb-420bad31388e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0682989-3595-4cd6-90a5-ab9111c278e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"67068120-e3ff-4817-bd06-44feb7479a35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09e8b-8a85-4997-bdea-75d754e6ee67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2f0d9ead-f47f-4cb3-92a4-0a5ad23cd831","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fdb6a25f-6bd8-4c65-9b9b-b460ba7673a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:35.751 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.774+0000 7f50357fa640 10 librados: watch_flush enter 2024-03-29T11:22:35.751 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.774+0000 7f50357fa640 10 librados: watch_flush exit 2024-03-29T11:22:35.751 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.774+0000 7f50357fa640 20 client.4288.objecter shutdown clearing up homeless session... 2024-03-29T11:22:35.751 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.774+0000 7f50357fa640 10 client.4288.objecter successfully canceled tick 2024-03-29T11:22:35.751 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.774+0000 7f50357fa640 1 -- 172.21.15.114:0/120142017 >> v1:172.21.15.114:6816/4038755055 conn(0x7f503c06de20 legacy=0x7f503c0702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:35.751 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.774+0000 7f50357fa640 10 monclient: shutdown 2024-03-29T11:22:35.752 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.774+0000 7f50357fa640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:35.752 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.774+0000 7f50357fa640 1 -- 172.21.15.114:0/120142017 >> v1:172.21.15.176:6789/0 conn(0x7f5048007c80 legacy=0x7f50480c2280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:35.752 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.774+0000 7f50357fa640 1 -- 172.21.15.114:0/120142017 shutdown_connections 2024-03-29T11:22:35.752 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.774+0000 7f50357fa640 1 -- 172.21.15.114:0/120142017 >> 172.21.15.114:0/120142017 conn(0x7f5048006a10 msgr2=0x7f5048157aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:35.752 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.774+0000 7f50357fa640 1 -- 172.21.15.114:0/120142017 shutdown_connections 2024-03-29T11:22:35.752 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.774+0000 7f50357fa640 1 -- 172.21.15.114:0/120142017 wait complete. 2024-03-29T11:22:35.752 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:35.774+0000 7f50357fa640 1 librados: shutdown 2024-03-29T11:22:35.768 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:42.574 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:42.734 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.756+0000 7f3c7973a640 10 monclient: get_monmap_and_config 2024-03-29T11:22:42.734 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.756+0000 7f3c7973a640 10 monclient: build_initial_monmap 2024-03-29T11:22:42.735 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.756+0000 7f3c7973a640 10 monclient: monmap: 2024-03-29T11:22:42.735 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:22:42.735 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:22:42.735 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:22:42.757922+0000 2024-03-29T11:22:42.735 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:22:42.757922+0000 2024-03-29T11:22:42.735 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:42.735 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:22:42.735 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:22:42.735 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T11:22:42.735 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T11:22:42.735 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:22:42.736 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.758+0000 7f3c7973a640 1 Processor -- start 2024-03-29T11:22:42.736 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.758+0000 7f3c7973a640 1 -- start start 2024-03-29T11:22:42.736 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.758+0000 7f3c7973a640 10 monclient: init 2024-03-29T11:22:42.737 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.759+0000 7f3c7973a640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:42.737 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.759+0000 7f3c7973a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:22:42.737 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.759+0000 7f3c7973a640 10 monclient(hunting): picked mon.noname-b con 0x7f3c740ae250 addr v1:172.21.15.114:6790/0 2024-03-29T11:22:42.737 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.759+0000 7f3c7973a640 10 monclient(hunting): picked mon.noname-a con 0x7f3c7413f970 addr v1:172.21.15.114:6789/0 2024-03-29T11:22:42.737 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.759+0000 7f3c7973a640 10 monclient(hunting): picked mon.noname-c con 0x7f3c74142800 addr v1:172.21.15.176:6789/0 2024-03-29T11:22:42.737 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.759+0000 7f3c7973a640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c740afe00 con 0x7f3c7413f970 2024-03-29T11:22:42.737 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.759+0000 7f3c7973a640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c74142c10 con 0x7f3c74142800 2024-03-29T11:22:42.737 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.759+0000 7f3c7973a640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c740afb20 con 0x7f3c740ae250 2024-03-29T11:22:42.737 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.759+0000 7f3c7973a640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:42.738 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.759+0000 7f3c7973a640 10 monclient(hunting): authenticate will time out at 1032.941478s 2024-03-29T11:22:42.743 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.764+0000 7f3c72ffd640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f3c740ae250 0x7f3c7413f260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:38035/0 (socket says 172.21.15.114:38035) 2024-03-29T11:22:42.743 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.764+0000 7f3c72ffd640 1 -- 172.21.15.114:0/1040426597 learned_addr learned my addr 172.21.15.114:0/1040426597 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:22:42.743 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.765+0000 7f3c71ffb640 1 -- 172.21.15.114:0/1040426597 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1968462343 0 0) 0x7f3c740afb20 con 0x7f3c740ae250 2024-03-29T11:22:42.743 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.765+0000 7f3c71ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:22:42.743 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.765+0000 7f3c71ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:42.743 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.765+0000 7f3c71ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:42.743 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.765+0000 7f3c71ffb640 10 monclient(hunting): my global_id is 4286 2024-03-29T11:22:42.743 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.765+0000 7f3c71ffb640 1 -- 172.21.15.114:0/1040426597 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c60003520 con 0x7f3c740ae250 2024-03-29T11:22:42.744 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.765+0000 7f3c71ffb640 1 -- 172.21.15.114:0/1040426597 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 728950269 0 0) 0x7f3c740afe00 con 0x7f3c7413f970 2024-03-29T11:22:42.744 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.765+0000 7f3c71ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:22:42.744 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.765+0000 7f3c71ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:42.744 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.765+0000 7f3c71ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:42.744 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.765+0000 7f3c71ffb640 10 monclient(hunting): my global_id is 4299 2024-03-29T11:22:42.744 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.765+0000 7f3c71ffb640 1 -- 172.21.15.114:0/1040426597 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c740afb20 con 0x7f3c7413f970 2024-03-29T11:22:42.744 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.765+0000 7f3c71ffb640 1 -- 172.21.15.114:0/1040426597 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2001202989 0 0) 0x7f3c60003520 con 0x7f3c740ae250 2024-03-29T11:22:42.744 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.765+0000 7f3c71ffb640 1 -- 172.21.15.114:0/1040426597 >> v1:172.21.15.176:6789/0 conn(0x7f3c74142800 legacy=0x7f3c7414da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:22:42.744 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.765+0000 7f3c71ffb640 1 -- 172.21.15.114:0/1040426597 >> v1:172.21.15.114:6789/0 conn(0x7f3c7413f970 legacy=0x7f3c741420f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:42.744 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.765+0000 7f3c71ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:22:42.744 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.765+0000 7f3c71ffb640 1 monclient: found mon.noname-b 2024-03-29T11:22:42.744 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.765+0000 7f3c71ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:42.744 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.765+0000 7f3c71ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.114:6790/0 2024-03-29T11:22:42.744 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.765+0000 7f3c71ffb640 1 -- 172.21.15.114:0/1040426597 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c740aab80 con 0x7f3c740ae250 2024-03-29T11:22:42.744 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.765+0000 7f3c71ffb640 10 monclient: _finish_auth 0 2024-03-29T11:22:42.744 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.765+0000 7f3c71ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:22:42.745 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.765+0000 7f3c71ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:42.745 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.766+0000 7f3c71ffb640 1 -- 172.21.15.114:0/1040426597 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f3c5c001690 con 0x7f3c740ae250 2024-03-29T11:22:42.745 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.766+0000 7f3c71ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:42.745 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.766+0000 7f3c71ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:22:42.745 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.766+0000 7f3c71ffb640 10 monclient: dump: 2024-03-29T11:22:42.745 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:22:42.745 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:22:42.745 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:22:42.745 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:22:42.745 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:42.745 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:22:42.745 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:22:42.745 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:22:42.745 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:22:42.745 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:22:42.745 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.766+0000 7f3c71ffb640 1 -- 172.21.15.114:0/1040426597 <== mon.2 v1:172.21.15.114:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3c5c0018e0 con 0x7f3c740ae250 2024-03-29T11:22:42.745 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.766+0000 7f3c71ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:42.746 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.766+0000 7f3c71ffb640 1 -- 172.21.15.114:0/1040426597 <== mon.2 v1:172.21.15.114:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f3c5c001c70 con 0x7f3c740ae250 2024-03-29T11:22:42.746 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.766+0000 7f3c71ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:42.746 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.766+0000 7f3c71ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:22:42.746 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.766+0000 7f3c71ffb640 10 monclient: dump: 2024-03-29T11:22:42.746 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:22:42.746 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:22:42.746 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:22:42.746 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:22:42.746 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:42.746 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:22:42.746 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:22:42.746 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:22:42.746 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:22:42.746 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:22:42.747 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.766+0000 7f3c7973a640 5 monclient: authenticate success, global_id 4286 2024-03-29T11:22:42.747 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.766+0000 7f3c7973a640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:42.747 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.766+0000 7f3c7973a640 4 set_mon_vals no callback set 2024-03-29T11:22:42.748 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.767+0000 7f3c7973a640 10 monclient: shutdown 2024-03-29T11:22:42.748 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.767+0000 7f3c7973a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:42.748 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.767+0000 7f3c7973a640 1 -- 172.21.15.114:0/1040426597 >> v1:172.21.15.114:6790/0 conn(0x7f3c740ae250 legacy=0x7f3c7413f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:42.748 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.767+0000 7f3c7973a640 1 -- 172.21.15.114:0/1040426597 shutdown_connections 2024-03-29T11:22:42.748 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.767+0000 7f3c7973a640 1 -- 172.21.15.114:0/1040426597 >> 172.21.15.114:0/1040426597 conn(0x7f3c740b35e0 msgr2=0x7f3c740b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:42.748 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.768+0000 7f3c7973a640 1 -- 172.21.15.114:0/1040426597 shutdown_connections 2024-03-29T11:22:42.748 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.768+0000 7f3c7973a640 1 -- 172.21.15.114:0/1040426597 wait complete. 2024-03-29T11:22:42.748 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.769+0000 7f3c7973a640 10 monclient: build_initial_monmap 2024-03-29T11:22:42.748 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.769+0000 7f3c7973a640 10 monclient: monmap: 2024-03-29T11:22:42.748 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:22:42.749 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:42.749 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T11:22:42.749 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T11:22:42.749 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:42.749 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:22:42.749 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:22:42.749 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T11:22:42.749 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T11:22:42.749 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:22:42.749 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.769+0000 7f3c7973a640 1 librados: starting msgr at 2024-03-29T11:22:42.749 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.769+0000 7f3c7973a640 1 librados: starting objecter 2024-03-29T11:22:42.749 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.769+0000 7f3c7973a640 1 Processor -- start 2024-03-29T11:22:42.749 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.770+0000 7f3c7973a640 1 -- start start 2024-03-29T11:22:42.750 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.770+0000 7f3c7973a640 1 librados: setting wanted keys 2024-03-29T11:22:42.750 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.770+0000 7f3c7973a640 1 librados: calling monclient init 2024-03-29T11:22:42.750 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.770+0000 7f3c7973a640 10 monclient: init 2024-03-29T11:22:42.750 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.771+0000 7f3c7973a640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:42.750 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.771+0000 7f3c7973a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:22:42.750 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.771+0000 7f3c7973a640 10 monclient(hunting): picked mon.noname-c con 0x7f3c740ae250 addr v1:172.21.15.114:6790/0 2024-03-29T11:22:42.750 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.771+0000 7f3c7973a640 10 monclient(hunting): picked mon.noname-a con 0x7f3c74142800 addr v1:172.21.15.114:6789/0 2024-03-29T11:22:42.750 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.771+0000 7f3c7973a640 10 monclient(hunting): picked mon.noname-b con 0x7f3c74139f80 addr v1:172.21.15.176:6789/0 2024-03-29T11:22:42.750 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.771+0000 7f3c7973a640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c74143410 con 0x7f3c74142800 2024-03-29T11:22:42.750 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.771+0000 7f3c7973a640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c74139330 con 0x7f3c74139f80 2024-03-29T11:22:42.751 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.771+0000 7f3c7973a640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c740ad1e0 con 0x7f3c740ae250 2024-03-29T11:22:42.751 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.771+0000 7f3c7973a640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:42.751 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.771+0000 7f3c7973a640 10 monclient(hunting): authenticate will time out at 1032.953302s 2024-03-29T11:22:42.753 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.776+0000 7f3c72ffd640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f3c740ae250 0x7f3c74139870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:38129/0 (socket says 172.21.15.114:38129) 2024-03-29T11:22:42.753 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.776+0000 7f3c72ffd640 1 -- 172.21.15.114:0/1906665323 learned_addr learned my addr 172.21.15.114:0/1906665323 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:22:42.753 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.776+0000 7f3c6b7fe640 10 client.?.objecter ms_handle_connect 0x7f3c740ae250 2024-03-29T11:22:42.753 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.776+0000 7f3c6b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:42.753 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.776+0000 7f3c6b7fe640 1 -- 172.21.15.114:0/1906665323 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2259432404 0 0) 0x7f3c740ad1e0 con 0x7f3c740ae250 2024-03-29T11:22:42.753 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.776+0000 7f3c6b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:22:42.753 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.776+0000 7f3c6b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:42.754 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.776+0000 7f3c6b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:42.754 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.777+0000 7f3c6b7fe640 10 monclient(hunting): my global_id is 4289 2024-03-29T11:22:42.754 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.777+0000 7f3c6b7fe640 1 -- 172.21.15.114:0/1906665323 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c54004eb0 con 0x7f3c740ae250 2024-03-29T11:22:42.757 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.778+0000 7f3c6b7fe640 10 client.?.objecter ms_handle_connect 0x7f3c74142800 2024-03-29T11:22:42.757 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.779+0000 7f3c6b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:42.757 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.779+0000 7f3c6b7fe640 1 -- 172.21.15.114:0/1906665323 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3707254668 0 0) 0x7f3c74143410 con 0x7f3c74142800 2024-03-29T11:22:42.757 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.779+0000 7f3c6b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:22:42.757 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.779+0000 7f3c6b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:42.757 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.779+0000 7f3c6b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:42.757 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.779+0000 7f3c6b7fe640 10 monclient(hunting): my global_id is 4302 2024-03-29T11:22:42.757 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.779+0000 7f3c6b7fe640 1 -- 172.21.15.114:0/1906665323 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c740ad1e0 con 0x7f3c74142800 2024-03-29T11:22:42.757 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.779+0000 7f3c6b7fe640 1 -- 172.21.15.114:0/1906665323 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2973626191 0 0) 0x7f3c54004eb0 con 0x7f3c740ae250 2024-03-29T11:22:42.757 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.779+0000 7f3c6b7fe640 1 -- 172.21.15.114:0/1906665323 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3c74143410 con 0x7f3c740ae250 2024-03-29T11:22:42.757 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.779+0000 7f3c6b7fe640 1 -- 172.21.15.114:0/1906665323 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f3c5c001bf0 con 0x7f3c740ae250 2024-03-29T11:22:42.757 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.779+0000 7f3c6b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:42.758 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.779+0000 7f3c6b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:22:42.758 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.779+0000 7f3c6b7fe640 10 monclient(hunting): dump: 2024-03-29T11:22:42.758 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:22:42.758 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:22:42.758 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:22:42.758 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:22:42.758 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:42.758 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:22:42.758 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:22:42.758 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:22:42.758 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:22:42.758 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:22:42.758 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.779+0000 7f3c6b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:42.758 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.780+0000 7f3c6b7fe640 1 -- 172.21.15.114:0/1906665323 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3390050075 0 0) 0x7f3c740ad1e0 con 0x7f3c74142800 2024-03-29T11:22:42.759 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.780+0000 7f3c6b7fe640 1 -- 172.21.15.114:0/1906665323 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3c54004eb0 con 0x7f3c74142800 2024-03-29T11:22:42.759 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.780+0000 7f3c6b7fe640 1 -- 172.21.15.114:0/1906665323 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f3c6c001880 con 0x7f3c74142800 2024-03-29T11:22:42.759 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.780+0000 7f3c6b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:42.759 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.780+0000 7f3c6b7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:22:42.759 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.780+0000 7f3c6b7fe640 10 monclient(hunting): dump: 2024-03-29T11:22:42.759 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:22:42.759 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:22:42.759 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:22:42.759 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:22:42.759 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:42.759 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:22:42.759 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:22:42.759 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:22:42.759 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:22:42.759 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:22:42.761 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.780+0000 7f3c6b7fe640 1 -- 172.21.15.114:0/1906665323 <== mon.2 v1:172.21.15.114:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3131233332 0 0) 0x7f3c74143410 con 0x7f3c740ae250 2024-03-29T11:22:42.762 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.780+0000 7f3c6b7fe640 1 -- 172.21.15.114:0/1906665323 >> v1:172.21.15.176:6789/0 conn(0x7f3c74139f80 legacy=0x7f3c741ef5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:42.762 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.780+0000 7f3c6b7fe640 1 -- 172.21.15.114:0/1906665323 >> v1:172.21.15.114:6789/0 conn(0x7f3c74142800 legacy=0x7f3c74138670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:42.762 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.780+0000 7f3c6b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:22:42.762 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.780+0000 7f3c6b7fe640 1 monclient: found mon.c 2024-03-29T11:22:42.762 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.780+0000 7f3c6b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:42.762 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.780+0000 7f3c6b7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:22:42.762 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.780+0000 7f3c6b7fe640 1 -- 172.21.15.114:0/1906665323 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c7413c730 con 0x7f3c740ae250 2024-03-29T11:22:42.762 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.780+0000 7f3c6b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:22:42.762 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.780+0000 7f3c6b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:22:42.762 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.780+0000 7f3c6b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:42.762 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.780+0000 7f3c7973a640 5 monclient: authenticate success, global_id 4289 2024-03-29T11:22:42.762 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.780+0000 7f3c7973a640 10 monclient: _renew_subs 2024-03-29T11:22:42.762 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.780+0000 7f3c7973a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:22:42.763 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.780+0000 7f3c7973a640 1 -- 172.21.15.114:0/1906665323 --> v1:172.21.15.114:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c74142090 con 0x7f3c740ae250 2024-03-29T11:22:42.763 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.780+0000 7f3c7973a640 10 client.4289.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:42.763 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.780+0000 7f3c7973a640 10 monclient: _renew_subs 2024-03-29T11:22:42.763 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.780+0000 7f3c7973a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:22:42.763 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.780+0000 7f3c7973a640 1 -- 172.21.15.114:0/1906665323 --> v1:172.21.15.114:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c7413bca0 con 0x7f3c740ae250 2024-03-29T11:22:42.763 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.780+0000 7f3c7973a640 1 librados: init done 2024-03-29T11:22:42.763 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.782+0000 7f3c6b7fe640 1 -- 172.21.15.114:0/1906665323 <== mon.2 v1:172.21.15.114:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3c5c001d70 con 0x7f3c740ae250 2024-03-29T11:22:42.763 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.782+0000 7f3c6b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:42.763 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.782+0000 7f3c6b7fe640 1 -- 172.21.15.114:0/1906665323 <== mon.2 v1:172.21.15.114:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f3c5c004d90 con 0x7f3c740ae250 2024-03-29T11:22:42.763 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.782+0000 7f3c6b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:42.763 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.783+0000 7f3c6b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:22:42.763 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.783+0000 7f3c6b7fe640 10 monclient: dump: 2024-03-29T11:22:42.763 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:22:42.763 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:22:42.763 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:22:42.763 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:22:42.763 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:42.764 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:22:42.764 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:22:42.764 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:22:42.764 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:22:42.764 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:22:42.764 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.783+0000 7f3c717fa640 4 set_mon_vals no callback set 2024-03-29T11:22:42.764 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.784+0000 7f3c7973a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:42.764 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.784+0000 7f3c7973a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:42.764 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.784+0000 7f3c7973a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:22:42.764 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.784+0000 7f3c7973a640 1 -- 172.21.15.114:0/1906665323 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c58005080 con 0x7f3c740ae250 2024-03-29T11:22:42.764 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.785+0000 7f3c6b7fe640 1 -- 172.21.15.114:0/1906665323 <== mon.2 v1:172.21.15.114:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1594292461 0 0) 0x7f3c5c01ad60 con 0x7f3c740ae250 2024-03-29T11:22:42.765 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.785+0000 7f3c6b7fe640 1 -- 172.21.15.114:0/1906665323 <== mon.2 v1:172.21.15.114:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3238987313 0 0) 0x7f3c5c084b50 con 0x7f3c740ae250 2024-03-29T11:22:42.765 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.787+0000 7f3c727fc640 10 monclient: get_auth_request con 0x7f3c5406de90 auth_method 0 2024-03-29T11:22:42.765 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.788+0000 7f3c6b7fe640 10 client.4289.objecter ms_dispatch 0x7f3c74000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:22:42.765 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.788+0000 7f3c6b7fe640 3 client.4289.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:22:42.765 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.788+0000 7f3c6b7fe640 3 client.4289.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:22:42.765 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.788+0000 7f3c6b7fe640 20 client.4289.objecter dump_active .. 0 homeless 2024-03-29T11:22:42.765 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.788+0000 7f3c6b7fe640 10 client.4289.objecter ms_handle_connect 0x7f3c5406de90 2024-03-29T11:22:42.774 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.795+0000 7f3c6b7fe640 1 -- 172.21.15.114:0/1906665323 <== mon.2 v1:172.21.15.114:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3c5c050430 con 0x7f3c740ae250 2024-03-29T11:22:42.774 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.795+0000 7f3c6b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:42.774 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:42.795+0000 7f3c6b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:43.103 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:43.126+0000 7f3c7973a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:43.103 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:43.126+0000 7f3c7973a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:43.103 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:43.126+0000 7f3c7973a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:22:43.103 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:43.126+0000 7f3c7973a640 1 -- 172.21.15.114:0/1906665323 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3c58005330 con 0x7f3c740ae250 2024-03-29T11:22:43.104 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:43.127+0000 7f3c6b7fe640 1 -- 172.21.15.114:0/1906665323 <== mon.2 v1:172.21.15.114:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2858224613) 0x7f3c5c052360 con 0x7f3c740ae250 2024-03-29T11:22:43.104 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:43.127+0000 7f3c6b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:43.104 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:43.127+0000 7f3c6b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:43.105 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T11:22:43.105 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"b0de2b9b-c45a-44c7-b6ee-e83b39a1e108","created":"2024-03-29T11:20:49.420660+0000","modified":"2024-03-29T11:20:51.700167+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:51.700167+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a64785d3-6b11-499a-b706-cc827ab96a74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5882dadb-e7a8-4604-ba9a-418013c106c8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d7e8c409-bc66-4e4a-bbdb-420bad31388e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0682989-3595-4cd6-90a5-ab9111c278e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"67068120-e3ff-4817-bd06-44feb7479a35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09e8b-8a85-4997-bdea-75d754e6ee67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2f0d9ead-f47f-4cb3-92a4-0a5ad23cd831","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fdb6a25f-6bd8-4c65-9b9b-b460ba7673a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:43.107 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:43.130+0000 7f3c7973a640 10 librados: watch_flush enter 2024-03-29T11:22:43.108 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:43.130+0000 7f3c7973a640 10 librados: watch_flush exit 2024-03-29T11:22:43.108 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:43.131+0000 7f3c7973a640 20 client.4289.objecter shutdown clearing up homeless session... 2024-03-29T11:22:43.108 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:43.131+0000 7f3c7973a640 10 client.4289.objecter successfully canceled tick 2024-03-29T11:22:43.109 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:43.132+0000 7f3c7973a640 1 -- 172.21.15.114:0/1906665323 >> v1:172.21.15.114:6816/4038755055 conn(0x7f3c5406de90 legacy=0x7f3c54070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:43.110 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:43.132+0000 7f3c7973a640 10 monclient: shutdown 2024-03-29T11:22:43.110 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:43.132+0000 7f3c7973a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:43.110 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:43.132+0000 7f3c7973a640 1 -- 172.21.15.114:0/1906665323 >> v1:172.21.15.114:6790/0 conn(0x7f3c740ae250 legacy=0x7f3c74139870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:43.110 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:43.132+0000 7f3c7973a640 1 -- 172.21.15.114:0/1906665323 shutdown_connections 2024-03-29T11:22:43.110 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:43.132+0000 7f3c7973a640 1 -- 172.21.15.114:0/1906665323 >> 172.21.15.114:0/1906665323 conn(0x7f3c740b35e0 msgr2=0x7f3c7413e360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:43.110 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:43.133+0000 7f3c7973a640 1 -- 172.21.15.114:0/1906665323 shutdown_connections 2024-03-29T11:22:43.110 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:43.133+0000 7f3c7973a640 1 -- 172.21.15.114:0/1906665323 wait complete. 2024-03-29T11:22:43.114 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:43.134+0000 7f3c7973a640 1 librados: shutdown 2024-03-29T11:22:43.131 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:49.937 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:50.087 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.109+0000 7f021b474640 10 monclient: get_monmap_and_config 2024-03-29T11:22:50.088 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.109+0000 7f021b474640 10 monclient: build_initial_monmap 2024-03-29T11:22:50.088 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.109+0000 7f021b474640 10 monclient: monmap: 2024-03-29T11:22:50.088 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:22:50.088 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:22:50.088 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:22:50.110583+0000 2024-03-29T11:22:50.088 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:22:50.110583+0000 2024-03-29T11:22:50.088 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:50.088 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:22:50.088 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:22:50.088 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T11:22:50.088 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T11:22:50.088 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:22:50.089 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.112+0000 7f021b474640 1 Processor -- start 2024-03-29T11:22:50.093 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.114+0000 7f021b474640 1 -- start start 2024-03-29T11:22:50.093 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.114+0000 7f021b474640 10 monclient: init 2024-03-29T11:22:50.093 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.115+0000 7f021b474640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:50.094 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.115+0000 7f021b474640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:22:50.094 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.115+0000 7f021b474640 10 monclient(hunting): picked mon.noname-b con 0x7f02140a9ac0 addr v1:172.21.15.114:6790/0 2024-03-29T11:22:50.094 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.115+0000 7f021b474640 10 monclient(hunting): picked mon.noname-a con 0x7f021413ee90 addr v1:172.21.15.114:6789/0 2024-03-29T11:22:50.094 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.115+0000 7f021b474640 10 monclient(hunting): picked mon.noname-c con 0x7f02140af900 addr v1:172.21.15.176:6789/0 2024-03-29T11:22:50.094 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.115+0000 7f021b474640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02140afd10 con 0x7f021413ee90 2024-03-29T11:22:50.094 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.115+0000 7f021b474640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02140af6f0 con 0x7f02140af900 2024-03-29T11:22:50.094 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.115+0000 7f021b474640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02140aac80 con 0x7f02140a9ac0 2024-03-29T11:22:50.094 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.115+0000 7f021b474640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:50.094 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.115+0000 7f021b474640 10 monclient(hunting): authenticate will time out at 1040.297933s 2024-03-29T11:22:50.096 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.118+0000 7f0219210640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f02140a9ac0 0x7f02140adfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:60925/0 (socket says 172.21.15.114:60925) 2024-03-29T11:22:50.096 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.118+0000 7f0219210640 1 -- 172.21.15.114:0/1676376545 learned_addr learned my addr 172.21.15.114:0/1676376545 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:22:50.100 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.122+0000 7f020bfff640 1 -- 172.21.15.114:0/1676376545 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 322037644 0 0) 0x7f02140af6f0 con 0x7f02140af900 2024-03-29T11:22:50.100 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.122+0000 7f020bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:22:50.100 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.122+0000 7f020bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:50.100 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.122+0000 7f020bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:50.100 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.122+0000 7f020bfff640 10 monclient(hunting): my global_id is 4291 2024-03-29T11:22:50.100 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.122+0000 7f020bfff640 1 -- 172.21.15.114:0/1676376545 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0200003520 con 0x7f02140af900 2024-03-29T11:22:50.101 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.122+0000 7f020bfff640 1 -- 172.21.15.114:0/1676376545 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1323589650 0 0) 0x7f02140aac80 con 0x7f02140a9ac0 2024-03-29T11:22:50.101 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.122+0000 7f020bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:22:50.101 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.122+0000 7f020bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:50.101 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.122+0000 7f020bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:50.101 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.122+0000 7f020bfff640 10 monclient(hunting): my global_id is 4292 2024-03-29T11:22:50.101 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.122+0000 7f020bfff640 1 -- 172.21.15.114:0/1676376545 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02140af6f0 con 0x7f02140a9ac0 2024-03-29T11:22:50.101 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.122+0000 7f020bfff640 1 -- 172.21.15.114:0/1676376545 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 145428475 0 0) 0x7f0200003520 con 0x7f02140af900 2024-03-29T11:22:50.101 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.122+0000 7f020bfff640 1 -- 172.21.15.114:0/1676376545 >> v1:172.21.15.114:6790/0 conn(0x7f02140a9ac0 legacy=0x7f02140adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:50.101 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.122+0000 7f020bfff640 1 -- 172.21.15.114:0/1676376545 >> v1:172.21.15.114:6789/0 conn(0x7f021413ee90 legacy=0x7f021413b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:50.101 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.122+0000 7f020bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:22:50.101 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.122+0000 7f020bfff640 1 monclient: found mon.noname-c 2024-03-29T11:22:50.101 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.122+0000 7f020bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:50.101 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.123+0000 7f020bfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.176:6789/0 2024-03-29T11:22:50.101 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.123+0000 7f020bfff640 1 -- 172.21.15.114:0/1676376545 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02140aaf60 con 0x7f02140af900 2024-03-29T11:22:50.102 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.123+0000 7f020bfff640 10 monclient: _finish_auth 0 2024-03-29T11:22:50.102 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.123+0000 7f020bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:22:50.102 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.123+0000 7f020bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:50.102 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.123+0000 7f020bfff640 1 -- 172.21.15.114:0/1676376545 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f020c001a10 con 0x7f02140af900 2024-03-29T11:22:50.102 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.123+0000 7f020bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:50.102 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.123+0000 7f020bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:22:50.102 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.123+0000 7f020bfff640 10 monclient: dump: 2024-03-29T11:22:50.102 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:22:50.102 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:22:50.102 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:22:50.102 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:22:50.102 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:50.102 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:22:50.102 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:22:50.103 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:22:50.103 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:22:50.103 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:22:50.103 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.123+0000 7f021b474640 5 monclient: authenticate success, global_id 4291 2024-03-29T11:22:50.103 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.123+0000 7f021b474640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:50.103 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.123+0000 7f020bfff640 1 -- 172.21.15.114:0/1676376545 <== mon.1 v1:172.21.15.176:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f020c001c30 con 0x7f02140af900 2024-03-29T11:22:50.103 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.123+0000 7f020bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:50.103 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.123+0000 7f020bfff640 1 -- 172.21.15.114:0/1676376545 <== mon.1 v1:172.21.15.176:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f020c001fc0 con 0x7f02140af900 2024-03-29T11:22:50.103 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.123+0000 7f020bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:50.103 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.123+0000 7f020bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:22:50.103 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.123+0000 7f020bfff640 10 monclient: dump: 2024-03-29T11:22:50.103 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:22:50.103 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:22:50.103 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:22:50.103 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:22:50.103 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:50.104 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:22:50.104 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:22:50.104 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:22:50.104 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:22:50.104 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:22:50.104 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.123+0000 7f021b474640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:50.104 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.123+0000 7f021b474640 4 set_mon_vals no callback set 2024-03-29T11:22:50.105 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.125+0000 7f021b474640 10 monclient: shutdown 2024-03-29T11:22:50.105 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.125+0000 7f021b474640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:50.105 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.125+0000 7f021b474640 1 -- 172.21.15.114:0/1676376545 >> v1:172.21.15.176:6789/0 conn(0x7f02140af900 legacy=0x7f0214149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:50.105 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.127+0000 7f021b474640 1 -- 172.21.15.114:0/1676376545 shutdown_connections 2024-03-29T11:22:50.105 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.127+0000 7f021b474640 1 -- 172.21.15.114:0/1676376545 >> 172.21.15.114:0/1676376545 conn(0x7f02140b35e0 msgr2=0x7f02140b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:50.105 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.128+0000 7f021b474640 1 -- 172.21.15.114:0/1676376545 shutdown_connections 2024-03-29T11:22:50.106 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.128+0000 7f021b474640 1 -- 172.21.15.114:0/1676376545 wait complete. 2024-03-29T11:22:50.107 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.129+0000 7f021b474640 10 monclient: build_initial_monmap 2024-03-29T11:22:50.107 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.129+0000 7f021b474640 10 monclient: monmap: 2024-03-29T11:22:50.107 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:22:50.108 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:50.108 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T11:22:50.108 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T11:22:50.108 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:50.108 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:22:50.108 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:22:50.108 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T11:22:50.108 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T11:22:50.108 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:22:50.108 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.130+0000 7f021b474640 1 librados: starting msgr at 2024-03-29T11:22:50.108 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.130+0000 7f021b474640 1 librados: starting objecter 2024-03-29T11:22:50.108 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.130+0000 7f021b474640 1 Processor -- start 2024-03-29T11:22:50.108 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.130+0000 7f021b474640 1 -- start start 2024-03-29T11:22:50.108 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.130+0000 7f021b474640 1 librados: setting wanted keys 2024-03-29T11:22:50.108 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.130+0000 7f021b474640 1 librados: calling monclient init 2024-03-29T11:22:50.109 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.130+0000 7f021b474640 10 monclient: init 2024-03-29T11:22:50.109 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.131+0000 7f021b474640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:50.109 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.131+0000 7f021b474640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:22:50.112 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.133+0000 7f021b474640 10 monclient(hunting): picked mon.noname-c con 0x7f02140a9ac0 addr v1:172.21.15.114:6790/0 2024-03-29T11:22:50.112 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.133+0000 7f021b474640 10 monclient(hunting): picked mon.noname-b con 0x7f02140af900 addr v1:172.21.15.176:6789/0 2024-03-29T11:22:50.112 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.133+0000 7f021b474640 10 monclient(hunting): picked mon.noname-a con 0x7f021413ee90 addr v1:172.21.15.114:6789/0 2024-03-29T11:22:50.112 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.133+0000 7f021b474640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02141d61e0 con 0x7f021413ee90 2024-03-29T11:22:50.112 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.133+0000 7f021b474640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0214142170 con 0x7f02140af900 2024-03-29T11:22:50.112 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.133+0000 7f021b474640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0214142300 con 0x7f02140a9ac0 2024-03-29T11:22:50.112 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.133+0000 7f021b474640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:50.112 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.133+0000 7f021b474640 10 monclient(hunting): authenticate will time out at 1040.315347s 2024-03-29T11:22:50.112 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.135+0000 7f0219210640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f02140a9ac0 0x7f0214141a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:32807/0 (socket says 172.21.15.114:32807) 2024-03-29T11:22:50.113 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.135+0000 7f0219210640 1 -- 172.21.15.114:0/2756221513 learned_addr learned my addr 172.21.15.114:0/2756221513 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:22:50.113 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.136+0000 7f0209ffb640 10 client.?.objecter ms_handle_connect 0x7f02140a9ac0 2024-03-29T11:22:50.113 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.136+0000 7f0209ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:50.114 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.137+0000 7f0209ffb640 10 client.?.objecter ms_handle_connect 0x7f021413ee90 2024-03-29T11:22:50.114 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.137+0000 7f0209ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:50.115 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.138+0000 7f0209ffb640 10 client.?.objecter ms_handle_connect 0x7f02140af900 2024-03-29T11:22:50.115 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.138+0000 7f0209ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:50.116 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.138+0000 7f0209ffb640 1 -- 172.21.15.114:0/2756221513 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2610507949 0 0) 0x7f02141d61e0 con 0x7f021413ee90 2024-03-29T11:22:50.116 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.138+0000 7f0209ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:22:50.116 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.138+0000 7f0209ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:50.116 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.138+0000 7f0209ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:50.116 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.138+0000 7f0209ffb640 10 monclient(hunting): my global_id is 4308 2024-03-29T11:22:50.117 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.139+0000 7f0209ffb640 1 -- 172.21.15.114:0/2756221513 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01f8004ee0 con 0x7f021413ee90 2024-03-29T11:22:50.117 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.139+0000 7f0209ffb640 1 -- 172.21.15.114:0/2756221513 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 830690453 0 0) 0x7f0214142170 con 0x7f02140af900 2024-03-29T11:22:50.117 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.139+0000 7f0209ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:22:50.117 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.139+0000 7f0209ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:50.117 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.139+0000 7f0209ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:50.117 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.139+0000 7f0209ffb640 10 monclient(hunting): my global_id is 4294 2024-03-29T11:22:50.117 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.139+0000 7f0209ffb640 1 -- 172.21.15.114:0/2756221513 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02141d61e0 con 0x7f02140af900 2024-03-29T11:22:50.117 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.139+0000 7f0209ffb640 1 -- 172.21.15.114:0/2756221513 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3726434935 0 0) 0x7f0214142300 con 0x7f02140a9ac0 2024-03-29T11:22:50.117 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.139+0000 7f0209ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:22:50.118 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.139+0000 7f0209ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:50.118 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.139+0000 7f0209ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:50.118 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.139+0000 7f0209ffb640 10 monclient(hunting): my global_id is 4295 2024-03-29T11:22:50.118 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.139+0000 7f0209ffb640 1 -- 172.21.15.114:0/2756221513 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0214142170 con 0x7f02140a9ac0 2024-03-29T11:22:50.118 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.139+0000 7f0209ffb640 1 -- 172.21.15.114:0/2756221513 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4073219518 0 0) 0x7f0214142170 con 0x7f02140a9ac0 2024-03-29T11:22:50.118 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.139+0000 7f0209ffb640 1 -- 172.21.15.114:0/2756221513 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0214142300 con 0x7f02140a9ac0 2024-03-29T11:22:50.118 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.139+0000 7f0209ffb640 1 -- 172.21.15.114:0/2756221513 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3525834213 0 0) 0x7f01f8004ee0 con 0x7f021413ee90 2024-03-29T11:22:50.118 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.139+0000 7f0209ffb640 1 -- 172.21.15.114:0/2756221513 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f01f8006ac0 con 0x7f021413ee90 2024-03-29T11:22:50.118 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.139+0000 7f0209ffb640 1 -- 172.21.15.114:0/2756221513 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f020c001950 con 0x7f021413ee90 2024-03-29T11:22:50.118 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.139+0000 7f0209ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:50.118 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.139+0000 7f0209ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:22:50.118 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.139+0000 7f0209ffb640 10 monclient(hunting): dump: 2024-03-29T11:22:50.119 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:22:50.119 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:22:50.119 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:22:50.119 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:22:50.119 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:50.119 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:22:50.119 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:22:50.119 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:22:50.119 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:22:50.119 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:22:50.119 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.139+0000 7f0209ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:50.119 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.139+0000 7f0209ffb640 1 -- 172.21.15.114:0/2756221513 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3461005334 0 0) 0x7f02141d61e0 con 0x7f02140af900 2024-03-29T11:22:50.119 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.139+0000 7f0209ffb640 1 -- 172.21.15.114:0/2756221513 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f01f8004ee0 con 0x7f02140af900 2024-03-29T11:22:50.119 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.139+0000 7f0209ffb640 1 -- 172.21.15.114:0/2756221513 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f0204001810 con 0x7f02140af900 2024-03-29T11:22:50.120 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.139+0000 7f0209ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:50.120 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.139+0000 7f0209ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:22:50.120 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.139+0000 7f0209ffb640 10 monclient(hunting): dump: 2024-03-29T11:22:50.120 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:22:50.120 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:22:50.120 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:22:50.120 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:22:50.120 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:50.120 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:22:50.120 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:22:50.120 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:22:50.120 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:22:50.120 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:22:50.120 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.139+0000 7f0209ffb640 1 -- 172.21.15.114:0/2756221513 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f0210001810 con 0x7f02140a9ac0 2024-03-29T11:22:50.120 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.139+0000 7f0209ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:50.121 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.139+0000 7f0209ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:22:50.121 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.139+0000 7f0209ffb640 10 monclient(hunting): dump: 2024-03-29T11:22:50.121 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:22:50.121 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:22:50.121 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:22:50.121 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:22:50.121 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:50.121 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:22:50.121 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:22:50.121 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:22:50.121 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:22:50.121 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:22:50.121 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.139+0000 7f0209ffb640 1 -- 172.21.15.114:0/2756221513 <== mon.0 v1:172.21.15.114:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 959404912 0 0) 0x7f01f8006ac0 con 0x7f021413ee90 2024-03-29T11:22:50.121 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.139+0000 7f0209ffb640 1 -- 172.21.15.114:0/2756221513 >> v1:172.21.15.114:6790/0 conn(0x7f02140a9ac0 legacy=0x7f0214141a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:50.122 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.139+0000 7f0209ffb640 1 -- 172.21.15.114:0/2756221513 >> v1:172.21.15.176:6789/0 conn(0x7f02140af900 legacy=0x7f02141d30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:50.122 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.139+0000 7f0209ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:22:50.122 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.139+0000 7f0209ffb640 1 monclient: found mon.a 2024-03-29T11:22:50.122 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.139+0000 7f0209ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:50.122 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.139+0000 7f0209ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T11:22:50.122 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.139+0000 7f0209ffb640 1 -- 172.21.15.114:0/2756221513 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0214142490 con 0x7f021413ee90 2024-03-29T11:22:50.122 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.139+0000 7f0209ffb640 10 monclient: _finish_auth 0 2024-03-29T11:22:50.122 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.139+0000 7f0209ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:22:50.122 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.139+0000 7f0209ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:50.122 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.140+0000 7f021b474640 5 monclient: authenticate success, global_id 4308 2024-03-29T11:22:50.122 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.140+0000 7f021b474640 10 monclient: _renew_subs 2024-03-29T11:22:50.122 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.140+0000 7f021b474640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T11:22:50.122 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.140+0000 7f021b474640 1 -- 172.21.15.114:0/2756221513 --> v1:172.21.15.114:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02141d3810 con 0x7f021413ee90 2024-03-29T11:22:50.122 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.140+0000 7f0209ffb640 1 -- 172.21.15.114:0/2756221513 <== mon.0 v1:172.21.15.114:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f020c001ab0 con 0x7f021413ee90 2024-03-29T11:22:50.123 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.140+0000 7f0209ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:50.123 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.140+0000 7f020b7fe640 4 set_mon_vals no callback set 2024-03-29T11:22:50.125 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.142+0000 7f0209ffb640 1 -- 172.21.15.114:0/2756221513 <== mon.0 v1:172.21.15.114:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f020c003dc0 con 0x7f021413ee90 2024-03-29T11:22:50.125 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.142+0000 7f0209ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:50.125 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.142+0000 7f0209ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:22:50.125 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.142+0000 7f0209ffb640 10 monclient: dump: 2024-03-29T11:22:50.125 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:22:50.125 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:22:50.125 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:22:50.125 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:22:50.125 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:50.125 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:22:50.125 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:22:50.125 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:22:50.126 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:22:50.126 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:22:50.126 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.142+0000 7f0209ffb640 1 -- 172.21.15.114:0/2756221513 <== mon.0 v1:172.21.15.114:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1594292461 0 0) 0x7f020c019e10 con 0x7f021413ee90 2024-03-29T11:22:50.126 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.142+0000 7f021b474640 10 client.4308.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:50.126 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.142+0000 7f021b474640 10 monclient: _renew_subs 2024-03-29T11:22:50.126 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.142+0000 7f021b474640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T11:22:50.126 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.142+0000 7f021b474640 1 -- 172.21.15.114:0/2756221513 --> v1:172.21.15.114:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0214138f30 con 0x7f021413ee90 2024-03-29T11:22:50.126 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.142+0000 7f021b474640 1 librados: init done 2024-03-29T11:22:50.126 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.142+0000 7f0209ffb640 1 -- 172.21.15.114:0/2756221513 <== mon.0 v1:172.21.15.114:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3238987313 0 0) 0x7f020c053cc0 con 0x7f021413ee90 2024-03-29T11:22:50.126 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.142+0000 7f0209ffb640 10 client.4308.objecter ms_dispatch 0x7f0214000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:22:50.126 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.142+0000 7f0209ffb640 3 client.4308.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:22:50.126 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.142+0000 7f0209ffb640 3 client.4308.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:22:50.126 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.142+0000 7f0209ffb640 20 client.4308.objecter dump_active .. 0 homeless 2024-03-29T11:22:50.126 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.143+0000 7f0219210640 10 monclient: get_auth_request con 0x7f01f8076ca0 auth_method 0 2024-03-29T11:22:50.127 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.143+0000 7f021b474640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:50.127 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.144+0000 7f0209ffb640 10 client.4308.objecter ms_handle_connect 0x7f01f8076ca0 2024-03-29T11:22:50.127 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.147+0000 7f021b474640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:50.127 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.147+0000 7f021b474640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T11:22:50.127 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.147+0000 7f021b474640 1 -- 172.21.15.114:0/2756221513 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0214075f10 con 0x7f021413ee90 2024-03-29T11:22:50.132 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.155+0000 7f0209ffb640 1 -- 172.21.15.114:0/2756221513 <== mon.0 v1:172.21.15.114:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f020c04f600 con 0x7f021413ee90 2024-03-29T11:22:50.132 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.155+0000 7f0209ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:50.132 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.155+0000 7f0209ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:50.460 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.483+0000 7f021b474640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:50.461 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.483+0000 7f021b474640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:50.461 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.483+0000 7f021b474640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T11:22:50.461 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.483+0000 7f021b474640 1 -- 172.21.15.114:0/2756221513 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f021406ab70 con 0x7f021413ee90 2024-03-29T11:22:50.461 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.484+0000 7f0209ffb640 1 -- 172.21.15.114:0/2756221513 <== mon.0 v1:172.21.15.114:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2858224613) 0x7f020c051700 con 0x7f021413ee90 2024-03-29T11:22:50.461 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.484+0000 7f0209ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:50.461 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.484+0000 7f0209ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:50.462 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T11:22:50.462 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"b0de2b9b-c45a-44c7-b6ee-e83b39a1e108","created":"2024-03-29T11:20:49.420660+0000","modified":"2024-03-29T11:20:51.700167+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:51.700167+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a64785d3-6b11-499a-b706-cc827ab96a74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5882dadb-e7a8-4604-ba9a-418013c106c8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d7e8c409-bc66-4e4a-bbdb-420bad31388e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0682989-3595-4cd6-90a5-ab9111c278e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"67068120-e3ff-4817-bd06-44feb7479a35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09e8b-8a85-4997-bdea-75d754e6ee67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2f0d9ead-f47f-4cb3-92a4-0a5ad23cd831","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fdb6a25f-6bd8-4c65-9b9b-b460ba7673a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:50.465 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.487+0000 7f021b474640 10 librados: watch_flush enter 2024-03-29T11:22:50.467 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.489+0000 7f021b474640 10 librados: watch_flush exit 2024-03-29T11:22:50.467 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.489+0000 7f021b474640 20 client.4308.objecter shutdown clearing up homeless session... 2024-03-29T11:22:50.467 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.489+0000 7f021b474640 10 client.4308.objecter successfully canceled tick 2024-03-29T11:22:50.468 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.491+0000 7f021b474640 1 -- 172.21.15.114:0/2756221513 >> v1:172.21.15.114:6816/4038755055 conn(0x7f01f8076ca0 legacy=0x7f01f8079160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:50.468 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.491+0000 7f021b474640 10 monclient: shutdown 2024-03-29T11:22:50.469 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.491+0000 7f021b474640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:50.469 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.491+0000 7f021b474640 1 -- 172.21.15.114:0/2756221513 >> v1:172.21.15.114:6789/0 conn(0x7f021413ee90 legacy=0x7f02141dd760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:50.469 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.491+0000 7f021b474640 1 -- 172.21.15.114:0/2756221513 shutdown_connections 2024-03-29T11:22:50.469 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.491+0000 7f021b474640 1 -- 172.21.15.114:0/2756221513 >> 172.21.15.114:0/2756221513 conn(0x7f02140b35e0 msgr2=0x7f02140ad1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:50.469 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.492+0000 7f021b474640 1 -- 172.21.15.114:0/2756221513 shutdown_connections 2024-03-29T11:22:50.469 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.492+0000 7f021b474640 1 -- 172.21.15.114:0/2756221513 wait complete. 2024-03-29T11:22:50.470 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:50.492+0000 7f021b474640 1 librados: shutdown 2024-03-29T11:22:50.486 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:57.293 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:57.451 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.473+0000 7f605d1f5640 10 monclient: get_monmap_and_config 2024-03-29T11:22:57.451 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.473+0000 7f605d1f5640 10 monclient: build_initial_monmap 2024-03-29T11:22:57.451 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.473+0000 7f605d1f5640 10 monclient: monmap: 2024-03-29T11:22:57.451 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:22:57.451 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:22:57.451 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:22:57.474224+0000 2024-03-29T11:22:57.451 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:22:57.474224+0000 2024-03-29T11:22:57.451 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:57.451 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:22:57.451 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:22:57.451 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T11:22:57.451 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T11:22:57.452 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:22:57.455 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.475+0000 7f605d1f5640 1 Processor -- start 2024-03-29T11:22:57.455 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.475+0000 7f605d1f5640 1 -- start start 2024-03-29T11:22:57.455 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.475+0000 7f605d1f5640 10 monclient: init 2024-03-29T11:22:57.455 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.475+0000 7f605d1f5640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:57.455 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.475+0000 7f605d1f5640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:22:57.455 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.475+0000 7f605d1f5640 10 monclient(hunting): picked mon.noname-b con 0x7f60580b2790 addr v1:172.21.15.114:6790/0 2024-03-29T11:22:57.455 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.475+0000 7f605d1f5640 10 monclient(hunting): picked mon.noname-c con 0x7f6058143c80 addr v1:172.21.15.176:6789/0 2024-03-29T11:22:57.455 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.476+0000 7f605d1f5640 10 monclient(hunting): picked mon.noname-a con 0x7f6058146e30 addr v1:172.21.15.114:6789/0 2024-03-29T11:22:57.455 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.476+0000 7f605d1f5640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6058146290 con 0x7f6058146e30 2024-03-29T11:22:57.456 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.476+0000 7f605d1f5640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60580b3970 con 0x7f6058143c80 2024-03-29T11:22:57.456 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.476+0000 7f605d1f5640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f605806ee80 con 0x7f60580b2790 2024-03-29T11:22:57.456 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.476+0000 7f605d1f5640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:57.456 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.476+0000 7f605d1f5640 10 monclient(hunting): authenticate will time out at 1047.658141s 2024-03-29T11:22:57.457 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.479+0000 7f6057fff640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f60580b2790 0x7f6058143570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:48503/0 (socket says 172.21.15.114:48503) 2024-03-29T11:22:57.457 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.479+0000 7f6057fff640 1 -- 172.21.15.114:0/3909716256 learned_addr learned my addr 172.21.15.114:0/3909716256 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:22:57.460 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.482+0000 7f6056ffd640 1 -- 172.21.15.114:0/3909716256 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3239885041 0 0) 0x7f605806ee80 con 0x7f60580b2790 2024-03-29T11:22:57.460 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.482+0000 7f6056ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:22:57.460 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.482+0000 7f6056ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:57.460 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.482+0000 7f6056ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:57.460 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.482+0000 7f6056ffd640 10 monclient(hunting): my global_id is 4298 2024-03-29T11:22:57.460 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.482+0000 7f6056ffd640 1 -- 172.21.15.114:0/3909716256 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6040003520 con 0x7f60580b2790 2024-03-29T11:22:57.466 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.485+0000 7f6056ffd640 1 -- 172.21.15.114:0/3909716256 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 768707773 0 0) 0x7f6040003520 con 0x7f60580b2790 2024-03-29T11:22:57.466 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.485+0000 7f6056ffd640 1 -- 172.21.15.114:0/3909716256 >> v1:172.21.15.176:6789/0 conn(0x7f6058143c80 legacy=0x7f6058146720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:57.466 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.485+0000 7f6056ffd640 1 -- 172.21.15.114:0/3909716256 >> v1:172.21.15.114:6789/0 conn(0x7f6058146e30 legacy=0x7f6058151bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:22:57.466 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.485+0000 7f6056ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:22:57.467 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.485+0000 7f6056ffd640 1 monclient: found mon.noname-b 2024-03-29T11:22:57.467 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.485+0000 7f6056ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:57.467 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.485+0000 7f6056ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.114:6790/0 2024-03-29T11:22:57.467 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.485+0000 7f6056ffd640 1 -- 172.21.15.114:0/3909716256 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f605806f160 con 0x7f60580b2790 2024-03-29T11:22:57.467 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.485+0000 7f6056ffd640 10 monclient: _finish_auth 0 2024-03-29T11:22:57.467 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.485+0000 7f6056ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:22:57.467 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.485+0000 7f6056ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:57.467 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.485+0000 7f6056ffd640 1 -- 172.21.15.114:0/3909716256 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f604c001690 con 0x7f60580b2790 2024-03-29T11:22:57.467 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.485+0000 7f605d1f5640 5 monclient: authenticate success, global_id 4298 2024-03-29T11:22:57.467 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.485+0000 7f605d1f5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:57.467 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.485+0000 7f6056ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:57.467 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.485+0000 7f6056ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:22:57.467 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.485+0000 7f6056ffd640 10 monclient: dump: 2024-03-29T11:22:57.467 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:22:57.468 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:22:57.468 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:22:57.468 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:22:57.468 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:57.468 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:22:57.468 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:22:57.468 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:22:57.468 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:22:57.468 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:22:57.468 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.485+0000 7f605d1f5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:57.468 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.486+0000 7f6056ffd640 1 -- 172.21.15.114:0/3909716256 <== mon.2 v1:172.21.15.114:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f604c001860 con 0x7f60580b2790 2024-03-29T11:22:57.468 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.486+0000 7f6056ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:57.468 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.486+0000 7f605d1f5640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:57.469 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.486+0000 7f605d1f5640 4 set_mon_vals no callback set 2024-03-29T11:22:57.469 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.487+0000 7f6056ffd640 1 -- 172.21.15.114:0/3909716256 <== mon.2 v1:172.21.15.114:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f604c001cb0 con 0x7f60580b2790 2024-03-29T11:22:57.469 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.487+0000 7f6056ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:57.469 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.487+0000 7f6056ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:22:57.469 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.487+0000 7f6056ffd640 10 monclient: dump: 2024-03-29T11:22:57.469 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:22:57.469 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:22:57.469 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:22:57.469 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:22:57.469 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:57.469 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:22:57.470 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:22:57.470 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:22:57.470 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:22:57.470 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:22:57.470 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.487+0000 7f605d1f5640 10 monclient: shutdown 2024-03-29T11:22:57.470 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.487+0000 7f605d1f5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:57.470 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.487+0000 7f605d1f5640 1 -- 172.21.15.114:0/3909716256 >> v1:172.21.15.114:6790/0 conn(0x7f60580b2790 legacy=0x7f6058143570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:57.470 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.487+0000 7f605d1f5640 1 -- 172.21.15.114:0/3909716256 shutdown_connections 2024-03-29T11:22:57.470 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.487+0000 7f605d1f5640 1 -- 172.21.15.114:0/3909716256 >> 172.21.15.114:0/3909716256 conn(0x7f605806d070 msgr2=0x7f6058066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:57.470 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.487+0000 7f605d1f5640 1 -- 172.21.15.114:0/3909716256 shutdown_connections 2024-03-29T11:22:57.470 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.487+0000 7f605d1f5640 1 -- 172.21.15.114:0/3909716256 wait complete. 2024-03-29T11:22:57.470 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.488+0000 7f605d1f5640 10 monclient: build_initial_monmap 2024-03-29T11:22:57.470 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.488+0000 7f605d1f5640 10 monclient: monmap: 2024-03-29T11:22:57.471 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:22:57.471 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:57.471 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T11:22:57.471 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T11:22:57.471 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:57.471 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:22:57.471 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:22:57.471 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T11:22:57.471 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T11:22:57.471 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:22:57.471 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.488+0000 7f605d1f5640 1 librados: starting msgr at 2024-03-29T11:22:57.471 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.488+0000 7f605d1f5640 1 librados: starting objecter 2024-03-29T11:22:57.471 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.489+0000 7f605d1f5640 1 Processor -- start 2024-03-29T11:22:57.474 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.489+0000 7f605d1f5640 1 -- start start 2024-03-29T11:22:57.474 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.489+0000 7f605d1f5640 1 librados: setting wanted keys 2024-03-29T11:22:57.475 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.489+0000 7f605d1f5640 1 librados: calling monclient init 2024-03-29T11:22:57.475 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.489+0000 7f605d1f5640 10 monclient: init 2024-03-29T11:22:57.475 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.490+0000 7f605d1f5640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:57.475 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.490+0000 7f605d1f5640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:22:57.475 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.490+0000 7f605d1f5640 10 monclient(hunting): picked mon.noname-b con 0x7f60580b2790 addr v1:172.21.15.176:6789/0 2024-03-29T11:22:57.475 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.490+0000 7f605d1f5640 10 monclient(hunting): picked mon.noname-a con 0x7f6058143c80 addr v1:172.21.15.114:6789/0 2024-03-29T11:22:57.475 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.490+0000 7f605d1f5640 10 monclient(hunting): picked mon.noname-c con 0x7f6058146e30 addr v1:172.21.15.114:6790/0 2024-03-29T11:22:57.475 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.490+0000 7f605d1f5640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60580b62e0 con 0x7f6058143c80 2024-03-29T11:22:57.475 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.490+0000 7f605d1f5640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60580b6830 con 0x7f60580b2790 2024-03-29T11:22:57.475 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.490+0000 7f605d1f5640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60581461d0 con 0x7f6058146e30 2024-03-29T11:22:57.475 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.490+0000 7f605d1f5640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:57.475 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.490+0000 7f605d1f5640 10 monclient(hunting): authenticate will time out at 1047.672479s 2024-03-29T11:22:57.476 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.495+0000 7f6057fff640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f60580b2790 0x7f60580b6fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.114:58401/0 (socket says 172.21.15.114:58401) 2024-03-29T11:22:57.476 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.495+0000 7f6057fff640 1 -- 172.21.15.114:0/1706880934 learned_addr learned my addr 172.21.15.114:0/1706880934 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:22:57.476 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.496+0000 7f6054ff9640 10 client.?.objecter ms_handle_connect 0x7f60580b2790 2024-03-29T11:22:57.476 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.496+0000 7f6054ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:57.476 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.496+0000 7f6054ff9640 1 -- 172.21.15.114:0/1706880934 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1180251123 0 0) 0x7f60580b6830 con 0x7f60580b2790 2024-03-29T11:22:57.476 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.496+0000 7f6054ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:22:57.476 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.496+0000 7f6054ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:57.476 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.496+0000 7f6054ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:57.476 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.496+0000 7f6054ff9640 10 monclient(hunting): my global_id is 4300 2024-03-29T11:22:57.476 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.497+0000 7f6054ff9640 1 -- 172.21.15.114:0/1706880934 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6044004eb0 con 0x7f60580b2790 2024-03-29T11:22:57.477 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.499+0000 7f6054ff9640 1 -- 172.21.15.114:0/1706880934 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3743757114 0 0) 0x7f6044004eb0 con 0x7f60580b2790 2024-03-29T11:22:57.477 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.499+0000 7f6054ff9640 1 -- 172.21.15.114:0/1706880934 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f60580b6830 con 0x7f60580b2790 2024-03-29T11:22:57.477 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.499+0000 7f6054ff9640 1 -- 172.21.15.114:0/1706880934 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f604c001b70 con 0x7f60580b2790 2024-03-29T11:22:57.477 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.499+0000 7f6054ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:57.477 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.499+0000 7f6054ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:22:57.477 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.499+0000 7f6054ff9640 10 monclient(hunting): dump: 2024-03-29T11:22:57.477 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:22:57.477 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:22:57.477 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:22:57.477 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:22:57.477 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:57.477 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:22:57.478 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:22:57.478 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:22:57.478 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:22:57.478 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:22:57.478 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.499+0000 7f6054ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:57.478 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.499+0000 7f6054ff9640 1 -- 172.21.15.114:0/1706880934 <== mon.1 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2913133145 0 0) 0x7f60580b6830 con 0x7f60580b2790 2024-03-29T11:22:57.479 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.500+0000 7f6054ff9640 1 -- 172.21.15.114:0/1706880934 >> v1:172.21.15.114:6790/0 conn(0x7f6058146e30 legacy=0x7f60581e5b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:57.479 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.500+0000 7f6054ff9640 1 -- 172.21.15.114:0/1706880934 >> v1:172.21.15.114:6789/0 conn(0x7f6058143c80 legacy=0x7f6058145ac0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:22:57.479 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.501+0000 7f6054ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:22:57.479 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.501+0000 7f6054ff9640 1 monclient: found mon.b 2024-03-29T11:22:57.479 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.501+0000 7f6054ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:57.479 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.501+0000 7f6054ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:22:57.479 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.501+0000 7f6054ff9640 1 -- 172.21.15.114:0/1706880934 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60580b6550 con 0x7f60580b2790 2024-03-29T11:22:57.479 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.501+0000 7f6054ff9640 10 monclient: _finish_auth 0 2024-03-29T11:22:57.479 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.501+0000 7f6054ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:22:57.479 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.501+0000 7f6054ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:57.480 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.501+0000 7f605d1f5640 5 monclient: authenticate success, global_id 4300 2024-03-29T11:22:57.480 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.501+0000 7f605d1f5640 10 monclient: _renew_subs 2024-03-29T11:22:57.480 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.501+0000 7f605d1f5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:22:57.480 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.501+0000 7f605d1f5640 1 -- 172.21.15.114:0/1706880934 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60581453e0 con 0x7f60580b2790 2024-03-29T11:22:57.480 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.501+0000 7f605d1f5640 10 client.4300.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:57.480 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.501+0000 7f605d1f5640 10 monclient: _renew_subs 2024-03-29T11:22:57.481 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.501+0000 7f605d1f5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:22:57.481 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.501+0000 7f605d1f5640 1 -- 172.21.15.114:0/1706880934 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f60581e7450 con 0x7f60580b2790 2024-03-29T11:22:57.481 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.501+0000 7f605d1f5640 1 librados: init done 2024-03-29T11:22:57.481 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.502+0000 7f6054ff9640 1 -- 172.21.15.114:0/1706880934 <== mon.1 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f604c004bf0 con 0x7f60580b2790 2024-03-29T11:22:57.481 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.502+0000 7f6054ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:57.481 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.502+0000 7f6054ff9640 1 -- 172.21.15.114:0/1706880934 <== mon.1 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f604c004f00 con 0x7f60580b2790 2024-03-29T11:22:57.481 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.502+0000 7f6054ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:57.481 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.502+0000 7f6054ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:22:57.481 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.502+0000 7f60567fc640 4 set_mon_vals no callback set 2024-03-29T11:22:57.481 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.502+0000 7f6054ff9640 10 monclient: dump: 2024-03-29T11:22:57.481 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:22:57.481 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:22:57.481 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:22:57.481 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:22:57.481 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:57.481 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:22:57.482 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:22:57.482 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:22:57.482 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:22:57.482 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:22:57.482 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.503+0000 7f605d1f5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:57.483 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.504+0000 7f605d1f5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:57.483 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.504+0000 7f605d1f5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:22:57.483 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.504+0000 7f605d1f5640 1 -- 172.21.15.114:0/1706880934 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6038005080 con 0x7f60580b2790 2024-03-29T11:22:57.483 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.505+0000 7f6054ff9640 1 -- 172.21.15.114:0/1706880934 <== mon.1 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1594292461 0 0) 0x7f604c01aed0 con 0x7f60580b2790 2024-03-29T11:22:57.483 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.505+0000 7f6054ff9640 1 -- 172.21.15.114:0/1706880934 <== mon.1 v1:172.21.15.176:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3238987313 0 0) 0x7f604c084c50 con 0x7f60580b2790 2024-03-29T11:22:57.484 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.505+0000 7f6054ff9640 10 client.4300.objecter ms_dispatch 0x7f6058000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:22:57.484 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.505+0000 7f6054ff9640 3 client.4300.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:22:57.484 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.506+0000 7f6054ff9640 3 client.4300.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:22:57.484 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.506+0000 7f6054ff9640 20 client.4300.objecter dump_active .. 0 homeless 2024-03-29T11:22:57.484 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.506+0000 7f60577fe640 10 monclient: get_auth_request con 0x7f604406dc80 auth_method 0 2024-03-29T11:22:57.484 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.506+0000 7f6054ff9640 10 client.4300.objecter ms_handle_connect 0x7f604406dc80 2024-03-29T11:22:57.492 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.514+0000 7f6054ff9640 1 -- 172.21.15.114:0/1706880934 <== mon.1 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f604c0505b0 con 0x7f60580b2790 2024-03-29T11:22:57.492 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.514+0000 7f6054ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:57.492 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.514+0000 7f6054ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:57.812 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.834+0000 7f605d1f5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:57.812 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.834+0000 7f605d1f5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:57.812 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.834+0000 7f605d1f5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:22:57.812 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.834+0000 7f605d1f5640 1 -- 172.21.15.114:0/1706880934 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6038005330 con 0x7f60580b2790 2024-03-29T11:22:57.814 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.837+0000 7f6054ff9640 1 -- 172.21.15.114:0/1706880934 <== mon.1 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2858224613) 0x7f604c0524e0 con 0x7f60580b2790 2024-03-29T11:22:57.815 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.837+0000 7f6054ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:57.815 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.837+0000 7f6054ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:57.815 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T11:22:57.815 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"b0de2b9b-c45a-44c7-b6ee-e83b39a1e108","created":"2024-03-29T11:20:49.420660+0000","modified":"2024-03-29T11:20:51.700167+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:51.700167+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a64785d3-6b11-499a-b706-cc827ab96a74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5882dadb-e7a8-4604-ba9a-418013c106c8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d7e8c409-bc66-4e4a-bbdb-420bad31388e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0682989-3595-4cd6-90a5-ab9111c278e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"67068120-e3ff-4817-bd06-44feb7479a35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09e8b-8a85-4997-bdea-75d754e6ee67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2f0d9ead-f47f-4cb3-92a4-0a5ad23cd831","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fdb6a25f-6bd8-4c65-9b9b-b460ba7673a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:57.818 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.840+0000 7f605d1f5640 10 librados: watch_flush enter 2024-03-29T11:22:57.819 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.841+0000 7f605d1f5640 10 librados: watch_flush exit 2024-03-29T11:22:57.819 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.841+0000 7f605d1f5640 20 client.4300.objecter shutdown clearing up homeless session... 2024-03-29T11:22:57.819 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.841+0000 7f605d1f5640 10 client.4300.objecter successfully canceled tick 2024-03-29T11:22:57.819 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.841+0000 7f605d1f5640 1 -- 172.21.15.114:0/1706880934 >> v1:172.21.15.114:6816/4038755055 conn(0x7f604406dc80 legacy=0x7f6044070140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:57.819 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.841+0000 7f605d1f5640 10 monclient: shutdown 2024-03-29T11:22:57.819 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.841+0000 7f605d1f5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:57.819 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.841+0000 7f605d1f5640 1 -- 172.21.15.114:0/1706880934 >> v1:172.21.15.176:6789/0 conn(0x7f60580b2790 legacy=0x7f60580b6fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:57.819 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.841+0000 7f605d1f5640 1 -- 172.21.15.114:0/1706880934 shutdown_connections 2024-03-29T11:22:57.819 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.841+0000 7f605d1f5640 1 -- 172.21.15.114:0/1706880934 >> 172.21.15.114:0/1706880934 conn(0x7f605806d070 msgr2=0x7f6058140630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:57.820 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.842+0000 7f605d1f5640 1 -- 172.21.15.114:0/1706880934 shutdown_connections 2024-03-29T11:22:57.820 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.842+0000 7f605d1f5640 1 -- 172.21.15.114:0/1706880934 wait complete. 2024-03-29T11:22:57.820 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:22:57.842+0000 7f605d1f5640 1 librados: shutdown 2024-03-29T11:22:57.840 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:23:04.647 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:23:04.794 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.815+0000 7f8729ac5640 10 monclient: get_monmap_and_config 2024-03-29T11:23:04.794 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.815+0000 7f8729ac5640 10 monclient: build_initial_monmap 2024-03-29T11:23:04.794 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.815+0000 7f8729ac5640 10 monclient: monmap: 2024-03-29T11:23:04.794 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:23:04.794 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:23:04.794 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:23:04.816362+0000 2024-03-29T11:23:04.795 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:23:04.816362+0000 2024-03-29T11:23:04.795 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:04.795 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:23:04.795 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:23:04.795 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T11:23:04.795 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T11:23:04.795 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:23:04.795 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.816+0000 7f8729ac5640 1 Processor -- start 2024-03-29T11:23:04.796 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.816+0000 7f8729ac5640 1 -- start start 2024-03-29T11:23:04.796 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.816+0000 7f8729ac5640 10 monclient: init 2024-03-29T11:23:04.796 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.817+0000 7f8729ac5640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:04.797 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.817+0000 7f8729ac5640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:23:04.797 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.817+0000 7f8729ac5640 10 monclient(hunting): picked mon.noname-b con 0x7f87240a9750 addr v1:172.21.15.114:6790/0 2024-03-29T11:23:04.797 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.817+0000 7f8729ac5640 10 monclient(hunting): picked mon.noname-a con 0x7f8724149ac0 addr v1:172.21.15.114:6789/0 2024-03-29T11:23:04.797 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.817+0000 7f8729ac5640 10 monclient(hunting): picked mon.noname-c con 0x7f872414fe50 addr v1:172.21.15.176:6789/0 2024-03-29T11:23:04.797 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.817+0000 7f8729ac5640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87240aa4a0 con 0x7f8724149ac0 2024-03-29T11:23:04.797 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.817+0000 7f8729ac5640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f872414a680 con 0x7f872414fe50 2024-03-29T11:23:04.797 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.817+0000 7f87237fe640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f87240a9750 0x7f87241436f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:48612/0 (socket says 172.21.15.114:48612) 2024-03-29T11:23:04.797 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.817+0000 7f87237fe640 1 -- 172.21.15.114:0/2484795008 learned_addr learned my addr 172.21.15.114:0/2484795008 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:23:04.797 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.817+0000 7f8729ac5640 1 -- 172.21.15.114:0/2484795008 --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87240af290 con 0x7f87240a9750 2024-03-29T11:23:04.797 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.817+0000 7f8729ac5640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:04.797 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.817+0000 7f8729ac5640 10 monclient(hunting): authenticate will time out at 1055.000022s 2024-03-29T11:23:04.797 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.818+0000 7f87227fc640 1 -- 172.21.15.114:0/2484795008 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 975961479 0 0) 0x7f87240af290 con 0x7f87240a9750 2024-03-29T11:23:04.797 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.818+0000 7f87227fc640 10 monclient(hunting): _negotiate 2024-03-29T11:23:04.797 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.818+0000 7f87227fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:04.797 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.818+0000 7f87227fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:04.798 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.818+0000 7f87227fc640 10 monclient(hunting): my global_id is 4301 2024-03-29T11:23:04.798 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.818+0000 7f87227fc640 1 -- 172.21.15.114:0/2484795008 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8710003520 con 0x7f87240a9750 2024-03-29T11:23:04.798 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.818+0000 7f87227fc640 1 -- 172.21.15.114:0/2484795008 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3993662326 0 0) 0x7f872414a680 con 0x7f872414fe50 2024-03-29T11:23:04.798 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.818+0000 7f87227fc640 10 monclient(hunting): _negotiate 2024-03-29T11:23:04.798 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.818+0000 7f87227fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:04.798 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.818+0000 7f87227fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:04.798 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.818+0000 7f87227fc640 10 monclient(hunting): my global_id is 4303 2024-03-29T11:23:04.798 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.818+0000 7f87227fc640 1 -- 172.21.15.114:0/2484795008 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87240af290 con 0x7f872414fe50 2024-03-29T11:23:04.798 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.818+0000 7f87227fc640 1 -- 172.21.15.114:0/2484795008 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 623766603 0 0) 0x7f87240aa4a0 con 0x7f8724149ac0 2024-03-29T11:23:04.798 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.818+0000 7f87227fc640 10 monclient(hunting): _negotiate 2024-03-29T11:23:04.798 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.818+0000 7f87227fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:04.798 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.818+0000 7f87227fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:04.798 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.818+0000 7f87227fc640 10 monclient(hunting): my global_id is 4311 2024-03-29T11:23:04.798 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.818+0000 7f87227fc640 1 -- 172.21.15.114:0/2484795008 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f872414a680 con 0x7f8724149ac0 2024-03-29T11:23:04.804 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.819+0000 7f87227fc640 1 -- 172.21.15.114:0/2484795008 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2584432260 0 0) 0x7f8710003520 con 0x7f87240a9750 2024-03-29T11:23:04.804 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.819+0000 7f87227fc640 1 -- 172.21.15.114:0/2484795008 >> v1:172.21.15.176:6789/0 conn(0x7f872414fe50 legacy=0x7f8724152240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:04.804 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.819+0000 7f87227fc640 1 -- 172.21.15.114:0/2484795008 >> v1:172.21.15.114:6789/0 conn(0x7f8724149ac0 legacy=0x7f8724149f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:04.804 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.819+0000 7f87227fc640 10 monclient: _finish_hunting 0 2024-03-29T11:23:04.804 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.819+0000 7f87227fc640 1 monclient: found mon.noname-b 2024-03-29T11:23:04.804 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.819+0000 7f87227fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:04.804 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.819+0000 7f87227fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.114:6790/0 2024-03-29T11:23:04.804 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.819+0000 7f87227fc640 1 -- 172.21.15.114:0/2484795008 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87240afb40 con 0x7f87240a9750 2024-03-29T11:23:04.804 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.819+0000 7f87227fc640 10 monclient: _finish_auth 0 2024-03-29T11:23:04.805 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.819+0000 7f87227fc640 10 monclient: _check_auth_tickets 2024-03-29T11:23:04.805 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.819+0000 7f87227fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:04.805 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.819+0000 7f87227fc640 1 -- 172.21.15.114:0/2484795008 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f8714001a10 con 0x7f87240a9750 2024-03-29T11:23:04.805 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.819+0000 7f87227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:04.805 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.819+0000 7f87227fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:23:04.805 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.819+0000 7f87227fc640 10 monclient: dump: 2024-03-29T11:23:04.805 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:23:04.805 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:23:04.805 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:23:04.805 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:23:04.805 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:04.805 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:23:04.805 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:23:04.805 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:23:04.805 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:23:04.806 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:23:04.806 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.819+0000 7f8729ac5640 5 monclient: authenticate success, global_id 4301 2024-03-29T11:23:04.806 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.819+0000 7f8729ac5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:04.806 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.819+0000 7f87227fc640 1 -- 172.21.15.114:0/2484795008 <== mon.2 v1:172.21.15.114:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8714001b70 con 0x7f87240a9750 2024-03-29T11:23:04.806 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.819+0000 7f87227fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:04.806 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.819+0000 7f87227fc640 1 -- 172.21.15.114:0/2484795008 <== mon.2 v1:172.21.15.114:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f8714001fc0 con 0x7f87240a9750 2024-03-29T11:23:04.806 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.819+0000 7f87227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:04.806 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.819+0000 7f87227fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:23:04.806 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.819+0000 7f87227fc640 10 monclient: dump: 2024-03-29T11:23:04.806 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:23:04.806 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:23:04.806 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:23:04.806 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:23:04.806 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:04.807 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:23:04.807 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:23:04.807 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:23:04.807 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:23:04.807 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:23:04.807 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.819+0000 7f8729ac5640 10 monclient: get_monmap_and_config success 2024-03-29T11:23:04.807 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.819+0000 7f8729ac5640 4 set_mon_vals no callback set 2024-03-29T11:23:04.807 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.820+0000 7f8729ac5640 10 monclient: shutdown 2024-03-29T11:23:04.807 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.820+0000 7f8729ac5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:04.807 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.820+0000 7f8729ac5640 1 -- 172.21.15.114:0/2484795008 >> v1:172.21.15.114:6790/0 conn(0x7f87240a9750 legacy=0x7f87241436f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:04.807 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.820+0000 7f8729ac5640 1 -- 172.21.15.114:0/2484795008 shutdown_connections 2024-03-29T11:23:04.807 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.820+0000 7f8729ac5640 1 -- 172.21.15.114:0/2484795008 >> 172.21.15.114:0/2484795008 conn(0x7f87240b35e0 msgr2=0x7f87240b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:04.807 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.820+0000 7f8729ac5640 1 -- 172.21.15.114:0/2484795008 shutdown_connections 2024-03-29T11:23:04.807 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.820+0000 7f8729ac5640 1 -- 172.21.15.114:0/2484795008 wait complete. 2024-03-29T11:23:04.808 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.821+0000 7f8729ac5640 10 monclient: build_initial_monmap 2024-03-29T11:23:04.808 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.821+0000 7f8729ac5640 10 monclient: monmap: 2024-03-29T11:23:04.808 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:23:04.808 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:23:04.808 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T11:23:04.808 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T11:23:04.808 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:04.808 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:23:04.808 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:23:04.808 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T11:23:04.808 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T11:23:04.808 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:23:04.808 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.822+0000 7f8729ac5640 1 librados: starting msgr at 2024-03-29T11:23:04.808 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.822+0000 7f8729ac5640 1 librados: starting objecter 2024-03-29T11:23:04.808 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.822+0000 7f8729ac5640 1 Processor -- start 2024-03-29T11:23:04.809 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.822+0000 7f8729ac5640 1 -- start start 2024-03-29T11:23:04.809 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.822+0000 7f8729ac5640 1 librados: setting wanted keys 2024-03-29T11:23:04.809 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.822+0000 7f8729ac5640 1 librados: calling monclient init 2024-03-29T11:23:04.809 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.822+0000 7f8729ac5640 10 monclient: init 2024-03-29T11:23:04.809 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.823+0000 7f8729ac5640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:04.809 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.823+0000 7f8729ac5640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:23:04.809 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.823+0000 7f8729ac5640 10 monclient(hunting): picked mon.noname-a con 0x7f87240a9750 addr v1:172.21.15.114:6789/0 2024-03-29T11:23:04.809 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.823+0000 7f8729ac5640 10 monclient(hunting): picked mon.noname-c con 0x7f8724149ac0 addr v1:172.21.15.114:6790/0 2024-03-29T11:23:04.809 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.823+0000 7f8729ac5640 10 monclient(hunting): picked mon.noname-b con 0x7f872414fe50 addr v1:172.21.15.176:6789/0 2024-03-29T11:23:04.809 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.823+0000 7f8729ac5640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f872406f7b0 con 0x7f87240a9750 2024-03-29T11:23:04.809 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.823+0000 7f8729ac5640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8724070df0 con 0x7f872414fe50 2024-03-29T11:23:04.809 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.823+0000 7f8729ac5640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8724071060 con 0x7f8724149ac0 2024-03-29T11:23:04.809 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.823+0000 7f8729ac5640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:04.809 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.823+0000 7f8729ac5640 10 monclient(hunting): authenticate will time out at 1055.005549s 2024-03-29T11:23:04.809 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.823+0000 7f87237fe640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f87240a9750 0x7f87240abb90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:40226/0 (socket says 172.21.15.114:40226) 2024-03-29T11:23:04.810 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.823+0000 7f87237fe640 1 -- 172.21.15.114:0/589205808 learned_addr learned my addr 172.21.15.114:0/589205808 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:23:04.810 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.823+0000 7f871bfff640 10 client.?.objecter ms_handle_connect 0x7f872414fe50 2024-03-29T11:23:04.810 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.823+0000 7f871bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:04.810 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.823+0000 7f871bfff640 10 client.?.objecter ms_handle_connect 0x7f87240a9750 2024-03-29T11:23:04.810 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.823+0000 7f871bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:04.810 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.824+0000 7f871bfff640 10 client.?.objecter ms_handle_connect 0x7f8724149ac0 2024-03-29T11:23:04.810 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.824+0000 7f871bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:04.810 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.824+0000 7f871bfff640 1 -- 172.21.15.114:0/589205808 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1762923084 0 0) 0x7f872406f7b0 con 0x7f87240a9750 2024-03-29T11:23:04.810 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.824+0000 7f871bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:23:04.810 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.824+0000 7f871bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:04.810 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.824+0000 7f871bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:04.810 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.824+0000 7f871bfff640 10 monclient(hunting): my global_id is 4314 2024-03-29T11:23:04.810 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.824+0000 7f871bfff640 1 -- 172.21.15.114:0/589205808 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8704004eb0 con 0x7f87240a9750 2024-03-29T11:23:04.811 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.824+0000 7f871bfff640 1 -- 172.21.15.114:0/589205808 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 341718999 0 0) 0x7f8724071060 con 0x7f8724149ac0 2024-03-29T11:23:04.811 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.824+0000 7f871bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:23:04.811 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.824+0000 7f871bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:04.811 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.824+0000 7f871bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:04.811 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.824+0000 7f871bfff640 10 monclient(hunting): my global_id is 4304 2024-03-29T11:23:04.811 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.824+0000 7f871bfff640 1 -- 172.21.15.114:0/589205808 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f872406f7b0 con 0x7f8724149ac0 2024-03-29T11:23:04.811 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.824+0000 7f871bfff640 1 -- 172.21.15.114:0/589205808 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3332061421 0 0) 0x7f8704004eb0 con 0x7f87240a9750 2024-03-29T11:23:04.811 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.824+0000 7f871bfff640 1 -- 172.21.15.114:0/589205808 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8704005490 con 0x7f87240a9750 2024-03-29T11:23:04.811 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.824+0000 7f871bfff640 1 -- 172.21.15.114:0/589205808 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f8714001f40 con 0x7f87240a9750 2024-03-29T11:23:04.811 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.824+0000 7f871bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:04.811 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.824+0000 7f871bfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:23:04.811 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.824+0000 7f871bfff640 10 monclient(hunting): dump: 2024-03-29T11:23:04.811 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:23:04.811 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:23:04.811 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:23:04.811 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:23:04.812 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:04.812 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:23:04.812 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:23:04.812 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:23:04.812 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:23:04.812 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:23:04.812 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.824+0000 7f871bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:23:04.812 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.825+0000 7f871bfff640 1 -- 172.21.15.114:0/589205808 <== mon.0 v1:172.21.15.114:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 782553084 0 0) 0x7f8704005490 con 0x7f87240a9750 2024-03-29T11:23:04.812 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.825+0000 7f871bfff640 1 -- 172.21.15.114:0/589205808 >> v1:172.21.15.114:6790/0 conn(0x7f8724149ac0 legacy=0x7f87240ac2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:04.812 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.825+0000 7f871bfff640 1 -- 172.21.15.114:0/589205808 >> v1:172.21.15.176:6789/0 conn(0x7f872414fe50 legacy=0x7f87241e6230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:04.812 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.825+0000 7f871bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:23:04.812 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.825+0000 7f871bfff640 1 monclient: found mon.a 2024-03-29T11:23:04.812 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.825+0000 7f871bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:04.812 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.825+0000 7f871bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T11:23:04.813 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.825+0000 7f871bfff640 1 -- 172.21.15.114:0/589205808 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8724071340 con 0x7f87240a9750 2024-03-29T11:23:04.813 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.825+0000 7f871bfff640 10 monclient: _finish_auth 0 2024-03-29T11:23:04.813 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.825+0000 7f871bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:23:04.813 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.825+0000 7f871bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:04.813 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.825+0000 7f8729ac5640 5 monclient: authenticate success, global_id 4314 2024-03-29T11:23:04.813 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.825+0000 7f8729ac5640 10 monclient: _renew_subs 2024-03-29T11:23:04.813 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.825+0000 7f8729ac5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T11:23:04.813 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.825+0000 7f8729ac5640 1 -- 172.21.15.114:0/589205808 --> v1:172.21.15.114:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8724071620 con 0x7f87240a9750 2024-03-29T11:23:04.813 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.825+0000 7f8729ac5640 10 client.4314.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:23:04.813 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.825+0000 7f8729ac5640 10 monclient: _renew_subs 2024-03-29T11:23:04.813 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.825+0000 7f8729ac5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T11:23:04.813 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.825+0000 7f8729ac5640 1 -- 172.21.15.114:0/589205808 --> v1:172.21.15.114:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8724072090 con 0x7f87240a9750 2024-03-29T11:23:04.813 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.825+0000 7f8729ac5640 1 librados: init done 2024-03-29T11:23:04.813 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.826+0000 7f871bfff640 1 -- 172.21.15.114:0/589205808 <== mon.0 v1:172.21.15.114:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8714001550 con 0x7f87240a9750 2024-03-29T11:23:04.814 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.826+0000 7f871bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:04.814 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.826+0000 7f8721ffb640 4 set_mon_vals no callback set 2024-03-29T11:23:04.814 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.826+0000 7f871bfff640 1 -- 172.21.15.114:0/589205808 <== mon.0 v1:172.21.15.114:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f8714004fb0 con 0x7f87240a9750 2024-03-29T11:23:04.814 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.826+0000 7f871bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:04.814 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.826+0000 7f871bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:23:04.814 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.826+0000 7f871bfff640 10 monclient: dump: 2024-03-29T11:23:04.814 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:23:04.814 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:23:04.814 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:23:04.814 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:23:04.814 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:04.814 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:23:04.814 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:23:04.814 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:23:04.814 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:23:04.815 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:23:04.816 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.827+0000 7f8729ac5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:04.816 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.827+0000 7f8729ac5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:04.816 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.827+0000 7f8729ac5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T11:23:04.816 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.827+0000 7f8729ac5640 1 -- 172.21.15.114:0/589205808 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8708005080 con 0x7f87240a9750 2024-03-29T11:23:04.816 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.828+0000 7f871bfff640 1 -- 172.21.15.114:0/589205808 <== mon.0 v1:172.21.15.114:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1594292461 0 0) 0x7f871401b000 con 0x7f87240a9750 2024-03-29T11:23:04.816 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.830+0000 7f8722ffd640 10 monclient: get_auth_request con 0x7f870406e090 auth_method 0 2024-03-29T11:23:04.816 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.830+0000 7f871bfff640 1 -- 172.21.15.114:0/589205808 <== mon.0 v1:172.21.15.114:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3238987313 0 0) 0x7f8714084d30 con 0x7f87240a9750 2024-03-29T11:23:04.816 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.830+0000 7f871bfff640 10 client.4314.objecter ms_dispatch 0x7f8724000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:23:04.816 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.830+0000 7f871bfff640 3 client.4314.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:23:04.816 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.830+0000 7f871bfff640 3 client.4314.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:23:04.816 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.830+0000 7f871bfff640 20 client.4314.objecter dump_active .. 0 homeless 2024-03-29T11:23:04.816 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.830+0000 7f871bfff640 10 client.4314.objecter ms_handle_connect 0x7f870406e090 2024-03-29T11:23:04.817 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.838+0000 7f871bfff640 1 -- 172.21.15.114:0/589205808 <== mon.0 v1:172.21.15.114:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8714050610 con 0x7f87240a9750 2024-03-29T11:23:04.817 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.838+0000 7f871bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:04.817 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:04.838+0000 7f871bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:23:05.186 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:05.208+0000 7f8729ac5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:05.187 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:05.208+0000 7f8729ac5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:05.187 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:05.208+0000 7f8729ac5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T11:23:05.187 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:05.208+0000 7f8729ac5640 1 -- 172.21.15.114:0/589205808 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8708005330 con 0x7f87240a9750 2024-03-29T11:23:05.187 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:05.209+0000 7f871bfff640 1 -- 172.21.15.114:0/589205808 <== mon.0 v1:172.21.15.114:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2858224613) 0x7f8714004e10 con 0x7f87240a9750 2024-03-29T11:23:05.187 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:05.209+0000 7f871bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:05.188 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:05.209+0000 7f871bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:23:05.188 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T11:23:05.188 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"b0de2b9b-c45a-44c7-b6ee-e83b39a1e108","created":"2024-03-29T11:20:49.420660+0000","modified":"2024-03-29T11:20:51.700167+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:51.700167+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a64785d3-6b11-499a-b706-cc827ab96a74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5882dadb-e7a8-4604-ba9a-418013c106c8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d7e8c409-bc66-4e4a-bbdb-420bad31388e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0682989-3595-4cd6-90a5-ab9111c278e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"67068120-e3ff-4817-bd06-44feb7479a35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09e8b-8a85-4997-bdea-75d754e6ee67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2f0d9ead-f47f-4cb3-92a4-0a5ad23cd831","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fdb6a25f-6bd8-4c65-9b9b-b460ba7673a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:23:05.191 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:05.213+0000 7f8729ac5640 10 librados: watch_flush enter 2024-03-29T11:23:05.191 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:05.213+0000 7f8729ac5640 10 librados: watch_flush exit 2024-03-29T11:23:05.191 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:05.213+0000 7f8729ac5640 20 client.4314.objecter shutdown clearing up homeless session... 2024-03-29T11:23:05.191 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:05.213+0000 7f8729ac5640 10 client.4314.objecter successfully canceled tick 2024-03-29T11:23:05.191 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:05.213+0000 7f8729ac5640 1 -- 172.21.15.114:0/589205808 >> v1:172.21.15.114:6816/4038755055 conn(0x7f870406e090 legacy=0x7f8704070550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:05.191 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:05.213+0000 7f8729ac5640 10 monclient: shutdown 2024-03-29T11:23:05.191 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:05.213+0000 7f8729ac5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:05.191 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:05.213+0000 7f8729ac5640 1 -- 172.21.15.114:0/589205808 >> v1:172.21.15.114:6789/0 conn(0x7f87240a9750 legacy=0x7f87240abb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:05.191 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:05.213+0000 7f8729ac5640 1 -- 172.21.15.114:0/589205808 shutdown_connections 2024-03-29T11:23:05.192 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:05.213+0000 7f8729ac5640 1 -- 172.21.15.114:0/589205808 >> 172.21.15.114:0/589205808 conn(0x7f87240b35e0 msgr2=0x7f87240adee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:05.192 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:05.214+0000 7f8729ac5640 1 -- 172.21.15.114:0/589205808 shutdown_connections 2024-03-29T11:23:05.192 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:05.214+0000 7f8729ac5640 1 -- 172.21.15.114:0/589205808 wait complete. 2024-03-29T11:23:05.192 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:05.214+0000 7f8729ac5640 1 librados: shutdown 2024-03-29T11:23:05.206 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:23:12.014 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:23:12.170 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.191+0000 7f549310e640 10 monclient: get_monmap_and_config 2024-03-29T11:23:12.170 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.191+0000 7f549310e640 10 monclient: build_initial_monmap 2024-03-29T11:23:12.170 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.191+0000 7f549310e640 10 monclient: monmap: 2024-03-29T11:23:12.170 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:23:12.171 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:23:12.171 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:23:12.192927+0000 2024-03-29T11:23:12.171 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:23:12.192927+0000 2024-03-29T11:23:12.171 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:12.171 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:23:12.171 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:23:12.171 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T11:23:12.171 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T11:23:12.171 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:23:12.172 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.192+0000 7f549310e640 1 Processor -- start 2024-03-29T11:23:12.172 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.193+0000 7f549310e640 1 -- start start 2024-03-29T11:23:12.172 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.193+0000 7f549310e640 10 monclient: init 2024-03-29T11:23:12.175 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.194+0000 7f549310e640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:12.175 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.194+0000 7f549310e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:23:12.175 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.194+0000 7f549310e640 10 monclient(hunting): picked mon.noname-b con 0x7f548c06d9d0 addr v1:172.21.15.114:6790/0 2024-03-29T11:23:12.175 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.194+0000 7f549310e640 10 monclient(hunting): picked mon.noname-c con 0x7f548c0b0860 addr v1:172.21.15.176:6789/0 2024-03-29T11:23:12.175 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.194+0000 7f549310e640 10 monclient(hunting): picked mon.noname-a con 0x7f548c0af8e0 addr v1:172.21.15.114:6789/0 2024-03-29T11:23:12.175 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.194+0000 7f548bfff640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f548c0b0860 0x7f548c146a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.114:45174/0 (socket says 172.21.15.114:45174) 2024-03-29T11:23:12.175 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.194+0000 7f548bfff640 1 -- 172.21.15.114:0/2330412383 learned_addr learned my addr 172.21.15.114:0/2330412383 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:23:12.175 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.194+0000 7f549310e640 1 -- 172.21.15.114:0/2330412383 --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f548c0aed10 con 0x7f548c0af8e0 2024-03-29T11:23:12.175 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.195+0000 7f549310e640 1 -- 172.21.15.114:0/2330412383 --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f548c0afd50 con 0x7f548c0b0860 2024-03-29T11:23:12.176 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.195+0000 7f549310e640 1 -- 172.21.15.114:0/2330412383 --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f548c0ab1b0 con 0x7f548c06d9d0 2024-03-29T11:23:12.176 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.195+0000 7f549310e640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:12.176 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.195+0000 7f549310e640 10 monclient(hunting): authenticate will time out at 1062.377343s 2024-03-29T11:23:12.176 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.195+0000 7f548b7fe640 1 -- 172.21.15.114:0/2330412383 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1313092426 0 0) 0x7f548c0afd50 con 0x7f548c0b0860 2024-03-29T11:23:12.176 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.195+0000 7f548b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:23:12.176 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.195+0000 7f548b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:12.176 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.195+0000 7f548b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:12.176 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.195+0000 7f548b7fe640 10 monclient(hunting): my global_id is 4309 2024-03-29T11:23:12.176 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.195+0000 7f548b7fe640 1 -- 172.21.15.114:0/2330412383 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f547c003520 con 0x7f548c0b0860 2024-03-29T11:23:12.176 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.195+0000 7f548b7fe640 1 -- 172.21.15.114:0/2330412383 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3024934243 0 0) 0x7f547c003520 con 0x7f548c0b0860 2024-03-29T11:23:12.176 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.195+0000 7f548b7fe640 1 -- 172.21.15.114:0/2330412383 >> v1:172.21.15.114:6790/0 conn(0x7f548c06d9d0 legacy=0x7f548c0af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:12.176 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.195+0000 7f548b7fe640 1 -- 172.21.15.114:0/2330412383 >> v1:172.21.15.114:6789/0 conn(0x7f548c0af8e0 legacy=0x7f548c14a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:12.176 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.195+0000 7f548b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:23:12.176 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.195+0000 7f548b7fe640 1 monclient: found mon.noname-c 2024-03-29T11:23:12.176 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.195+0000 7f548b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:12.177 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.195+0000 7f548b7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.176:6789/0 2024-03-29T11:23:12.177 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.195+0000 7f548b7fe640 1 -- 172.21.15.114:0/2330412383 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f548c0ab490 con 0x7f548c0b0860 2024-03-29T11:23:12.177 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.195+0000 7f548b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:23:12.177 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.195+0000 7f548b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:23:12.177 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.195+0000 7f548b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:12.177 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.195+0000 7f548b7fe640 1 -- 172.21.15.114:0/2330412383 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f5478001670 con 0x7f548c0b0860 2024-03-29T11:23:12.177 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.195+0000 7f548b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:12.177 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.195+0000 7f548b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:23:12.177 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.195+0000 7f548b7fe640 10 monclient: dump: 2024-03-29T11:23:12.177 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:23:12.177 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:23:12.177 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:23:12.177 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:23:12.177 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:12.177 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:23:12.178 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:23:12.178 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:23:12.178 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:23:12.178 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:23:12.178 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.195+0000 7f549310e640 5 monclient: authenticate success, global_id 4309 2024-03-29T11:23:12.178 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.195+0000 7f549310e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:12.178 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.195+0000 7f548b7fe640 1 -- 172.21.15.114:0/2330412383 <== mon.1 v1:172.21.15.176:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f54780017d0 con 0x7f548c0b0860 2024-03-29T11:23:12.178 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.195+0000 7f548b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:12.178 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.195+0000 7f548b7fe640 1 -- 172.21.15.114:0/2330412383 <== mon.1 v1:172.21.15.176:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f5478001c20 con 0x7f548c0b0860 2024-03-29T11:23:12.178 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.195+0000 7f548b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:12.178 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.195+0000 7f548b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:23:12.178 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.195+0000 7f548b7fe640 10 monclient: dump: 2024-03-29T11:23:12.178 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:23:12.178 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:23:12.179 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:23:12.179 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:23:12.179 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:12.179 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:23:12.179 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:23:12.179 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:23:12.179 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:23:12.179 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:23:12.179 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.196+0000 7f549310e640 10 monclient: get_monmap_and_config success 2024-03-29T11:23:12.179 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.196+0000 7f549310e640 4 set_mon_vals no callback set 2024-03-29T11:23:12.185 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.197+0000 7f549310e640 10 monclient: shutdown 2024-03-29T11:23:12.185 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.197+0000 7f549310e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:12.185 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.197+0000 7f549310e640 1 -- 172.21.15.114:0/2330412383 >> v1:172.21.15.176:6789/0 conn(0x7f548c0b0860 legacy=0x7f548c146a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:12.185 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.197+0000 7f549310e640 1 -- 172.21.15.114:0/2330412383 shutdown_connections 2024-03-29T11:23:12.186 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.197+0000 7f549310e640 1 -- 172.21.15.114:0/2330412383 >> 172.21.15.114:0/2330412383 conn(0x7f548c06b660 msgr2=0x7f548c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:12.186 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.198+0000 7f549310e640 1 -- 172.21.15.114:0/2330412383 shutdown_connections 2024-03-29T11:23:12.186 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.198+0000 7f549310e640 1 -- 172.21.15.114:0/2330412383 wait complete. 2024-03-29T11:23:12.186 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.199+0000 7f549310e640 10 monclient: build_initial_monmap 2024-03-29T11:23:12.186 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.199+0000 7f549310e640 10 monclient: monmap: 2024-03-29T11:23:12.186 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:23:12.186 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:23:12.186 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T11:23:12.186 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T11:23:12.186 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:12.186 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:23:12.186 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:23:12.186 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T11:23:12.186 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T11:23:12.187 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:23:12.187 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.200+0000 7f549310e640 1 librados: starting msgr at 2024-03-29T11:23:12.187 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.200+0000 7f549310e640 1 librados: starting objecter 2024-03-29T11:23:12.187 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.200+0000 7f549310e640 1 Processor -- start 2024-03-29T11:23:12.187 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.200+0000 7f549310e640 1 -- start start 2024-03-29T11:23:12.187 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.200+0000 7f549310e640 1 librados: setting wanted keys 2024-03-29T11:23:12.187 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.200+0000 7f549310e640 1 librados: calling monclient init 2024-03-29T11:23:12.187 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.200+0000 7f549310e640 10 monclient: init 2024-03-29T11:23:12.187 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.202+0000 7f549310e640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:12.187 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.202+0000 7f549310e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:23:12.187 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.202+0000 7f549310e640 10 monclient(hunting): picked mon.noname-b con 0x7f548c06d9d0 addr v1:172.21.15.176:6789/0 2024-03-29T11:23:12.187 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.202+0000 7f549310e640 10 monclient(hunting): picked mon.noname-a con 0x7f548c0af8e0 addr v1:172.21.15.114:6789/0 2024-03-29T11:23:12.187 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.202+0000 7f549310e640 10 monclient(hunting): picked mon.noname-c con 0x7f548c0b0860 addr v1:172.21.15.114:6790/0 2024-03-29T11:23:12.187 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.202+0000 7f549310e640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f548c138230 con 0x7f548c0af8e0 2024-03-29T11:23:12.188 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.202+0000 7f549310e640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f548c0b3ea0 con 0x7f548c06d9d0 2024-03-29T11:23:12.188 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.202+0000 7f549310e640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f548c0b4110 con 0x7f548c0b0860 2024-03-29T11:23:12.188 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.202+0000 7f549310e640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:12.188 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.202+0000 7f549310e640 10 monclient(hunting): authenticate will time out at 1062.384693s 2024-03-29T11:23:12.188 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.202+0000 7f5490eaa640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f548c06d9d0 0x7f548c1372f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.114:49572/0 (socket says 172.21.15.114:49572) 2024-03-29T11:23:12.188 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.202+0000 7f5490eaa640 1 -- 172.21.15.114:0/410064326 learned_addr learned my addr 172.21.15.114:0/410064326 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:23:12.188 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.203+0000 7f54897fa640 10 client.?.objecter ms_handle_connect 0x7f548c06d9d0 2024-03-29T11:23:12.188 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.203+0000 7f54897fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:12.188 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.203+0000 7f54897fa640 10 client.?.objecter ms_handle_connect 0x7f548c0b0860 2024-03-29T11:23:12.188 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.203+0000 7f54897fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:12.188 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.203+0000 7f54897fa640 10 client.?.objecter ms_handle_connect 0x7f548c0af8e0 2024-03-29T11:23:12.188 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.203+0000 7f54897fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:12.188 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.203+0000 7f54897fa640 1 -- 172.21.15.114:0/410064326 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1174767405 0 0) 0x7f548c0b3ea0 con 0x7f548c06d9d0 2024-03-29T11:23:12.188 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.203+0000 7f54897fa640 10 monclient(hunting): _negotiate 2024-03-29T11:23:12.188 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.203+0000 7f54897fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:12.189 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.203+0000 7f54897fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:12.189 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.203+0000 7f54897fa640 10 monclient(hunting): my global_id is 4312 2024-03-29T11:23:12.189 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.204+0000 7f54897fa640 1 -- 172.21.15.114:0/410064326 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5470004eb0 con 0x7f548c06d9d0 2024-03-29T11:23:12.189 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.204+0000 7f54897fa640 1 -- 172.21.15.114:0/410064326 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4105251616 0 0) 0x7f548c0b4110 con 0x7f548c0b0860 2024-03-29T11:23:12.189 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.204+0000 7f54897fa640 10 monclient(hunting): _negotiate 2024-03-29T11:23:12.189 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.204+0000 7f54897fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:12.189 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.204+0000 7f54897fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:12.189 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.204+0000 7f54897fa640 10 monclient(hunting): my global_id is 4310 2024-03-29T11:23:12.189 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.204+0000 7f54897fa640 1 -- 172.21.15.114:0/410064326 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f548c0b3ea0 con 0x7f548c0b0860 2024-03-29T11:23:12.189 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.204+0000 7f54897fa640 1 -- 172.21.15.114:0/410064326 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2913989973 0 0) 0x7f548c138230 con 0x7f548c0af8e0 2024-03-29T11:23:12.189 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.204+0000 7f54897fa640 10 monclient(hunting): _negotiate 2024-03-29T11:23:12.189 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.204+0000 7f54897fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:12.189 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.204+0000 7f54897fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:12.189 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.204+0000 7f54897fa640 10 monclient(hunting): my global_id is 4320 2024-03-29T11:23:12.190 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.204+0000 7f54897fa640 1 -- 172.21.15.114:0/410064326 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f548c0b4110 con 0x7f548c0af8e0 2024-03-29T11:23:12.190 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.204+0000 7f54897fa640 1 -- 172.21.15.114:0/410064326 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3703337978 0 0) 0x7f5470004eb0 con 0x7f548c06d9d0 2024-03-29T11:23:12.190 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.204+0000 7f54897fa640 1 -- 172.21.15.114:0/410064326 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5470005620 con 0x7f548c06d9d0 2024-03-29T11:23:12.190 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.204+0000 7f54897fa640 1 -- 172.21.15.114:0/410064326 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f5480001a30 con 0x7f548c06d9d0 2024-03-29T11:23:12.190 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.204+0000 7f54897fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:12.190 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.204+0000 7f54897fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:23:12.190 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.204+0000 7f54897fa640 10 monclient(hunting): dump: 2024-03-29T11:23:12.190 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:23:12.190 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:23:12.190 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:23:12.190 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:23:12.190 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:12.190 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:23:12.190 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:23:12.190 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:23:12.191 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:23:12.191 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:23:12.191 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.204+0000 7f54897fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:23:12.191 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.205+0000 7f54897fa640 1 -- 172.21.15.114:0/410064326 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3573327272 0 0) 0x7f548c0b4110 con 0x7f548c0af8e0 2024-03-29T11:23:12.191 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.205+0000 7f54897fa640 1 -- 172.21.15.114:0/410064326 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5470004eb0 con 0x7f548c0af8e0 2024-03-29T11:23:12.191 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.205+0000 7f54897fa640 1 -- 172.21.15.114:0/410064326 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f5478001c20 con 0x7f548c0af8e0 2024-03-29T11:23:12.191 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.205+0000 7f54897fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:12.191 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.205+0000 7f54897fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:23:12.191 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.205+0000 7f54897fa640 10 monclient(hunting): dump: 2024-03-29T11:23:12.191 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:23:12.191 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:23:12.191 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:23:12.191 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:23:12.191 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:12.192 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:23:12.192 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:23:12.192 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:23:12.192 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:23:12.192 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:23:12.192 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.205+0000 7f54897fa640 1 -- 172.21.15.114:0/410064326 <== mon.1 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 625353720 0 0) 0x7f5470005620 con 0x7f548c06d9d0 2024-03-29T11:23:12.192 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.205+0000 7f54897fa640 1 -- 172.21.15.114:0/410064326 >> v1:172.21.15.114:6790/0 conn(0x7f548c0b0860 legacy=0x7f548c0b1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:12.192 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.205+0000 7f54897fa640 1 -- 172.21.15.114:0/410064326 >> v1:172.21.15.114:6789/0 conn(0x7f548c0af8e0 legacy=0x7f548c0b14b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:12.192 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.205+0000 7f54897fa640 10 monclient: _finish_hunting 0 2024-03-29T11:23:12.192 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.205+0000 7f54897fa640 1 monclient: found mon.b 2024-03-29T11:23:12.192 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.205+0000 7f54897fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:12.192 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.205+0000 7f54897fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:23:12.192 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.205+0000 7f54897fa640 1 -- 172.21.15.114:0/410064326 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f548c138480 con 0x7f548c06d9d0 2024-03-29T11:23:12.192 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.205+0000 7f54897fa640 10 monclient: _finish_auth 0 2024-03-29T11:23:12.192 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.205+0000 7f54897fa640 10 monclient: _check_auth_tickets 2024-03-29T11:23:12.193 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.205+0000 7f54897fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:12.193 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.206+0000 7f549310e640 5 monclient: authenticate success, global_id 4312 2024-03-29T11:23:12.193 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.206+0000 7f549310e640 10 monclient: _renew_subs 2024-03-29T11:23:12.193 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.206+0000 7f549310e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:23:12.193 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.206+0000 7f549310e640 1 -- 172.21.15.114:0/410064326 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f548c0b3160 con 0x7f548c06d9d0 2024-03-29T11:23:12.193 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.206+0000 7f549310e640 10 client.4312.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:23:12.193 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.206+0000 7f549310e640 10 monclient: _renew_subs 2024-03-29T11:23:12.193 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.206+0000 7f549310e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:23:12.193 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.206+0000 7f549310e640 1 -- 172.21.15.114:0/410064326 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f548c1e12d0 con 0x7f548c06d9d0 2024-03-29T11:23:12.193 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.206+0000 7f549310e640 1 librados: init done 2024-03-29T11:23:12.193 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.206+0000 7f54897fa640 1 -- 172.21.15.114:0/410064326 <== mon.1 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5480001c70 con 0x7f548c06d9d0 2024-03-29T11:23:12.193 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.206+0000 7f54897fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:12.193 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.206+0000 7f54897fa640 1 -- 172.21.15.114:0/410064326 <== mon.1 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f5480003a60 con 0x7f548c06d9d0 2024-03-29T11:23:12.193 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.206+0000 7f54897fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:12.193 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.206+0000 7f54897fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:23:12.194 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.206+0000 7f54897fa640 10 monclient: dump: 2024-03-29T11:23:12.194 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:23:12.194 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:23:12.194 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:23:12.194 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:23:12.194 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:12.194 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:23:12.194 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:23:12.194 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:23:12.194 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:23:12.194 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:23:12.194 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.206+0000 7f548affd640 4 set_mon_vals no callback set 2024-03-29T11:23:12.194 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.207+0000 7f549310e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:12.194 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.207+0000 7f549310e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:12.194 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.207+0000 7f549310e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:23:12.195 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.207+0000 7f549310e640 1 -- 172.21.15.114:0/410064326 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f546c005080 con 0x7f548c06d9d0 2024-03-29T11:23:12.195 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.208+0000 7f54897fa640 1 -- 172.21.15.114:0/410064326 <== mon.1 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1594292461 0 0) 0x7f5480003bc0 con 0x7f548c06d9d0 2024-03-29T11:23:12.195 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.209+0000 7f54897fa640 1 -- 172.21.15.114:0/410064326 <== mon.1 v1:172.21.15.176:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3238987313 0 0) 0x7f5480084a60 con 0x7f548c06d9d0 2024-03-29T11:23:12.196 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.209+0000 7f54897fa640 10 client.4312.objecter ms_dispatch 0x7f548c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:23:12.196 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.210+0000 7f54897fa640 3 client.4312.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:23:12.196 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.210+0000 7f54897fa640 3 client.4312.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:23:12.196 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.210+0000 7f54897fa640 20 client.4312.objecter dump_active .. 0 homeless 2024-03-29T11:23:12.196 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.210+0000 7f548bfff640 10 monclient: get_auth_request con 0x7f547006e130 auth_method 0 2024-03-29T11:23:12.196 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.210+0000 7f54897fa640 10 client.4312.objecter ms_handle_connect 0x7f547006e130 2024-03-29T11:23:12.196 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.216+0000 7f54897fa640 1 -- 172.21.15.114:0/410064326 <== mon.1 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5480086e60 con 0x7f548c06d9d0 2024-03-29T11:23:12.196 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.216+0000 7f54897fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:12.196 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.216+0000 7f54897fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:23:12.550 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.572+0000 7f549310e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:12.551 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.572+0000 7f549310e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:12.551 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.572+0000 7f549310e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:23:12.551 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.572+0000 7f549310e640 1 -- 172.21.15.114:0/410064326 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f546c005330 con 0x7f548c06d9d0 2024-03-29T11:23:12.552 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.574+0000 7f54897fa640 1 -- 172.21.15.114:0/410064326 <== mon.1 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2858224613) 0x7f5480086e60 con 0x7f548c06d9d0 2024-03-29T11:23:12.552 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.574+0000 7f54897fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:12.553 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.574+0000 7f54897fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:23:12.553 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T11:23:12.553 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"b0de2b9b-c45a-44c7-b6ee-e83b39a1e108","created":"2024-03-29T11:20:49.420660+0000","modified":"2024-03-29T11:20:51.700167+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:51.700167+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a64785d3-6b11-499a-b706-cc827ab96a74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5882dadb-e7a8-4604-ba9a-418013c106c8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d7e8c409-bc66-4e4a-bbdb-420bad31388e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0682989-3595-4cd6-90a5-ab9111c278e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"67068120-e3ff-4817-bd06-44feb7479a35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09e8b-8a85-4997-bdea-75d754e6ee67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2f0d9ead-f47f-4cb3-92a4-0a5ad23cd831","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fdb6a25f-6bd8-4c65-9b9b-b460ba7673a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:23:12.556 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.577+0000 7f549310e640 10 librados: watch_flush enter 2024-03-29T11:23:12.556 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.577+0000 7f549310e640 10 librados: watch_flush exit 2024-03-29T11:23:12.556 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.577+0000 7f549310e640 20 client.4312.objecter shutdown clearing up homeless session... 2024-03-29T11:23:12.556 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.577+0000 7f549310e640 10 client.4312.objecter successfully canceled tick 2024-03-29T11:23:12.556 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.577+0000 7f549310e640 1 -- 172.21.15.114:0/410064326 >> v1:172.21.15.114:6816/4038755055 conn(0x7f547006e130 legacy=0x7f54700705f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:12.556 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.577+0000 7f549310e640 10 monclient: shutdown 2024-03-29T11:23:12.556 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.577+0000 7f549310e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:12.556 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.577+0000 7f549310e640 1 -- 172.21.15.114:0/410064326 >> v1:172.21.15.176:6789/0 conn(0x7f548c06d9d0 legacy=0x7f548c1372f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:12.556 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.577+0000 7f549310e640 1 -- 172.21.15.114:0/410064326 shutdown_connections 2024-03-29T11:23:12.556 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.577+0000 7f549310e640 1 -- 172.21.15.114:0/410064326 >> 172.21.15.114:0/410064326 conn(0x7f548c06b660 msgr2=0x7f548c12d120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:12.556 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.577+0000 7f549310e640 1 -- 172.21.15.114:0/410064326 shutdown_connections 2024-03-29T11:23:12.556 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.577+0000 7f549310e640 1 -- 172.21.15.114:0/410064326 wait complete. 2024-03-29T11:23:12.556 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:12.578+0000 7f549310e640 1 librados: shutdown 2024-03-29T11:23:12.570 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:23:19.377 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:23:19.513 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.533+0000 7f95a0625640 10 monclient: get_monmap_and_config 2024-03-29T11:23:19.513 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.533+0000 7f95a0625640 10 monclient: build_initial_monmap 2024-03-29T11:23:19.513 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.533+0000 7f95a0625640 10 monclient: monmap: 2024-03-29T11:23:19.513 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:23:19.514 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:23:19.514 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:23:19.534265+0000 2024-03-29T11:23:19.514 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:23:19.534265+0000 2024-03-29T11:23:19.514 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:19.514 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:23:19.514 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:23:19.515 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T11:23:19.515 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T11:23:19.515 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:23:19.515 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.534+0000 7f95a0625640 1 Processor -- start 2024-03-29T11:23:19.515 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.534+0000 7f95a0625640 1 -- start start 2024-03-29T11:23:19.515 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.534+0000 7f95a0625640 10 monclient: init 2024-03-29T11:23:19.534 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.535+0000 7f95a0625640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:19.534 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.535+0000 7f95a0625640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:23:19.534 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.535+0000 7f95a0625640 10 monclient(hunting): picked mon.noname-c con 0x7f95980a9750 addr v1:172.21.15.176:6789/0 2024-03-29T11:23:19.534 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.535+0000 7f95a0625640 10 monclient(hunting): picked mon.noname-a con 0x7f959813e1e0 addr v1:172.21.15.114:6789/0 2024-03-29T11:23:19.534 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.535+0000 7f95a0625640 10 monclient(hunting): picked mon.noname-b con 0x7f9598071c30 addr v1:172.21.15.114:6790/0 2024-03-29T11:23:19.534 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.535+0000 7f95a0625640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95980ab300 con 0x7f959813e1e0 2024-03-29T11:23:19.535 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.535+0000 7f95a0625640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95980ac2f0 con 0x7f95980a9750 2024-03-29T11:23:19.535 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.535+0000 7f95a0625640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95980ad8d0 con 0x7f9598071c30 2024-03-29T11:23:19.535 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.535+0000 7f95a0625640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:19.535 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.535+0000 7f95a0625640 10 monclient(hunting): authenticate will time out at 1069.717723s 2024-03-29T11:23:19.535 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.535+0000 7f959e3c1640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f95980a9750 0x7f959813dad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.114:33918/0 (socket says 172.21.15.114:33918) 2024-03-29T11:23:19.535 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.535+0000 7f959e3c1640 1 -- 172.21.15.114:0/2870154788 learned_addr learned my addr 172.21.15.114:0/2870154788 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:23:19.535 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.536+0000 7f959d3bf640 1 -- 172.21.15.114:0/2870154788 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3820837245 0 0) 0x7f95980ac2f0 con 0x7f95980a9750 2024-03-29T11:23:19.536 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.536+0000 7f959d3bf640 10 monclient(hunting): _negotiate 2024-03-29T11:23:19.536 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.536+0000 7f959d3bf640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:19.536 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.536+0000 7f959d3bf640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:19.536 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.536+0000 7f959d3bf640 10 monclient(hunting): my global_id is 4315 2024-03-29T11:23:19.536 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.536+0000 7f959d3bf640 1 -- 172.21.15.114:0/2870154788 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9584003440 con 0x7f95980a9750 2024-03-29T11:23:19.536 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.536+0000 7f959d3bf640 1 -- 172.21.15.114:0/2870154788 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1554805518 0 0) 0x7f95980ad8d0 con 0x7f9598071c30 2024-03-29T11:23:19.537 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.536+0000 7f959d3bf640 10 monclient(hunting): _negotiate 2024-03-29T11:23:19.537 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.536+0000 7f959d3bf640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:19.537 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.536+0000 7f959d3bf640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:19.537 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.536+0000 7f959d3bf640 10 monclient(hunting): my global_id is 4313 2024-03-29T11:23:19.537 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.536+0000 7f959d3bf640 1 -- 172.21.15.114:0/2870154788 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95980ac2f0 con 0x7f9598071c30 2024-03-29T11:23:19.537 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.536+0000 7f959d3bf640 1 -- 172.21.15.114:0/2870154788 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 194312762 0 0) 0x7f95980ac2f0 con 0x7f9598071c30 2024-03-29T11:23:19.538 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.536+0000 7f959d3bf640 1 -- 172.21.15.114:0/2870154788 >> v1:172.21.15.176:6789/0 conn(0x7f95980a9750 legacy=0x7f959813dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:19.538 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.536+0000 7f959d3bf640 1 -- 172.21.15.114:0/2870154788 >> v1:172.21.15.114:6789/0 conn(0x7f959813e1e0 legacy=0x7f95980ac710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:23:19.538 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.539+0000 7f959d3bf640 10 monclient: _finish_hunting 0 2024-03-29T11:23:19.538 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.539+0000 7f959d3bf640 1 monclient: found mon.noname-b 2024-03-29T11:23:19.538 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.539+0000 7f959d3bf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:19.538 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.539+0000 7f959d3bf640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.114:6790/0 2024-03-29T11:23:19.538 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.539+0000 7f959d3bf640 1 -- 172.21.15.114:0/2870154788 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95980ada60 con 0x7f9598071c30 2024-03-29T11:23:19.538 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.539+0000 7f959d3bf640 10 monclient: _finish_auth 0 2024-03-29T11:23:19.538 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.539+0000 7f959d3bf640 10 monclient: _check_auth_tickets 2024-03-29T11:23:19.539 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.539+0000 7f959d3bf640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:19.539 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.539+0000 7f959d3bf640 1 -- 172.21.15.114:0/2870154788 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f95900019f0 con 0x7f9598071c30 2024-03-29T11:23:19.539 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.539+0000 7f959d3bf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:19.539 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.539+0000 7f959d3bf640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:23:19.539 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.539+0000 7f959d3bf640 10 monclient: dump: 2024-03-29T11:23:19.539 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:23:19.539 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:23:19.539 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:23:19.539 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:23:19.539 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:19.539 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:23:19.539 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:23:19.539 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:23:19.539 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:23:19.540 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:23:19.540 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.539+0000 7f95a0625640 5 monclient: authenticate success, global_id 4313 2024-03-29T11:23:19.540 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.539+0000 7f95a0625640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:19.540 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.539+0000 7f959d3bf640 1 -- 172.21.15.114:0/2870154788 <== mon.2 v1:172.21.15.114:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9590001bd0 con 0x7f9598071c30 2024-03-29T11:23:19.540 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.539+0000 7f959d3bf640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:19.540 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.539+0000 7f95a0625640 10 monclient: get_monmap_and_config success 2024-03-29T11:23:19.540 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.539+0000 7f95a0625640 4 set_mon_vals no callback set 2024-03-29T11:23:19.540 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.539+0000 7f959d3bf640 1 -- 172.21.15.114:0/2870154788 <== mon.2 v1:172.21.15.114:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f9590002020 con 0x7f9598071c30 2024-03-29T11:23:19.540 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.539+0000 7f959d3bf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:19.540 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.539+0000 7f959d3bf640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:23:19.540 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.539+0000 7f959d3bf640 10 monclient: dump: 2024-03-29T11:23:19.540 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:23:19.540 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:23:19.540 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:23:19.541 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:23:19.541 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:19.541 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:23:19.541 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:23:19.541 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:23:19.541 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:23:19.541 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:23:19.541 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.540+0000 7f95a0625640 10 monclient: shutdown 2024-03-29T11:23:19.541 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.540+0000 7f95a0625640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:19.541 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.540+0000 7f95a0625640 1 -- 172.21.15.114:0/2870154788 >> v1:172.21.15.114:6790/0 conn(0x7f9598071c30 legacy=0x7f95980ace20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:19.541 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.540+0000 7f95a0625640 1 -- 172.21.15.114:0/2870154788 shutdown_connections 2024-03-29T11:23:19.541 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.540+0000 7f95a0625640 1 -- 172.21.15.114:0/2870154788 >> 172.21.15.114:0/2870154788 conn(0x7f959806b660 msgr2=0x7f959806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:19.541 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.540+0000 7f95a0625640 1 -- 172.21.15.114:0/2870154788 shutdown_connections 2024-03-29T11:23:19.541 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.540+0000 7f95a0625640 1 -- 172.21.15.114:0/2870154788 wait complete. 2024-03-29T11:23:19.542 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.541+0000 7f95a0625640 10 monclient: build_initial_monmap 2024-03-29T11:23:19.542 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.541+0000 7f95a0625640 10 monclient: monmap: 2024-03-29T11:23:19.542 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:23:19.542 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:23:19.542 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T11:23:19.542 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T11:23:19.542 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:19.542 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:23:19.542 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:23:19.542 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T11:23:19.542 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T11:23:19.542 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:23:19.542 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.541+0000 7f95a0625640 1 librados: starting msgr at 2024-03-29T11:23:19.542 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.541+0000 7f95a0625640 1 librados: starting objecter 2024-03-29T11:23:19.542 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.541+0000 7f95a0625640 1 Processor -- start 2024-03-29T11:23:19.543 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.542+0000 7f95a0625640 1 -- start start 2024-03-29T11:23:19.543 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.542+0000 7f95a0625640 1 librados: setting wanted keys 2024-03-29T11:23:19.543 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.542+0000 7f95a0625640 1 librados: calling monclient init 2024-03-29T11:23:19.543 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.542+0000 7f95a0625640 10 monclient: init 2024-03-29T11:23:19.543 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.542+0000 7f95a0625640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:19.543 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.542+0000 7f95a0625640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:23:19.543 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.543+0000 7f95a0625640 10 monclient(hunting): picked mon.noname-a con 0x7f9598071c30 addr v1:172.21.15.114:6789/0 2024-03-29T11:23:19.543 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.543+0000 7f959dbc0640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f95980a9750 0x7f95981d6820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:35444/0 (socket says 172.21.15.114:35444) 2024-03-29T11:23:19.543 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.543+0000 7f959dbc0640 1 -- 172.21.15.114:0/2613490337 learned_addr learned my addr 172.21.15.114:0/2613490337 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:23:19.543 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.543+0000 7f95a0625640 10 monclient(hunting): picked mon.noname-c con 0x7f95980a9750 addr v1:172.21.15.114:6790/0 2024-03-29T11:23:19.543 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.543+0000 7f95a0625640 10 monclient(hunting): picked mon.noname-b con 0x7f959813e1e0 addr v1:172.21.15.176:6789/0 2024-03-29T11:23:19.543 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.543+0000 7f95a0625640 1 -- 172.21.15.114:0/2613490337 --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9598140e40 con 0x7f9598071c30 2024-03-29T11:23:19.543 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.543+0000 7f95a0625640 1 -- 172.21.15.114:0/2613490337 --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f959813fa90 con 0x7f959813e1e0 2024-03-29T11:23:19.543 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.543+0000 7f95a0625640 1 -- 172.21.15.114:0/2613490337 --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9598142510 con 0x7f95980a9750 2024-03-29T11:23:19.544 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.543+0000 7f95a0625640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:19.544 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.543+0000 7f95a0625640 10 monclient(hunting): authenticate will time out at 1069.725867s 2024-03-29T11:23:19.544 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.543+0000 7f958effd640 10 client.?.objecter ms_handle_connect 0x7f95980a9750 2024-03-29T11:23:19.544 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.543+0000 7f958effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:19.544 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.544+0000 7f958effd640 1 -- 172.21.15.114:0/2613490337 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3167173064 0 0) 0x7f9598142510 con 0x7f95980a9750 2024-03-29T11:23:19.544 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.544+0000 7f958effd640 10 monclient(hunting): _negotiate 2024-03-29T11:23:19.544 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.544+0000 7f958effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:19.544 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.544+0000 7f958effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:19.544 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.544+0000 7f958effd640 10 monclient(hunting): my global_id is 4316 2024-03-29T11:23:19.544 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.544+0000 7f958effd640 1 -- 172.21.15.114:0/2613490337 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f957c004eb0 con 0x7f95980a9750 2024-03-29T11:23:19.544 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.544+0000 7f958effd640 10 client.?.objecter ms_handle_connect 0x7f959813e1e0 2024-03-29T11:23:19.544 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.544+0000 7f958effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:19.544 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.544+0000 7f958effd640 1 -- 172.21.15.114:0/2613490337 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3659213440 0 0) 0x7f957c004eb0 con 0x7f95980a9750 2024-03-29T11:23:19.544 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.544+0000 7f958effd640 1 -- 172.21.15.114:0/2613490337 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9598142510 con 0x7f95980a9750 2024-03-29T11:23:19.544 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.544+0000 7f958effd640 1 -- 172.21.15.114:0/2613490337 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f95880017d0 con 0x7f95980a9750 2024-03-29T11:23:19.545 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.544+0000 7f958effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:19.545 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.544+0000 7f958effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:23:19.545 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.544+0000 7f958effd640 10 monclient(hunting): dump: 2024-03-29T11:23:19.545 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:23:19.545 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:23:19.545 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:23:19.545 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:23:19.545 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:19.545 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:23:19.545 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:23:19.545 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:23:19.545 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:23:19.545 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:23:19.545 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.545+0000 7f958effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:23:19.545 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.545+0000 7f958effd640 10 client.?.objecter ms_handle_connect 0x7f9598071c30 2024-03-29T11:23:19.546 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.545+0000 7f958effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:19.546 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.545+0000 7f958effd640 1 -- 172.21.15.114:0/2613490337 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 166532573 0 0) 0x7f959813fa90 con 0x7f959813e1e0 2024-03-29T11:23:19.546 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.545+0000 7f958effd640 10 monclient(hunting): _negotiate 2024-03-29T11:23:19.546 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.545+0000 7f958effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:19.546 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.545+0000 7f958effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:19.546 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.545+0000 7f958effd640 10 monclient(hunting): my global_id is 4318 2024-03-29T11:23:19.546 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.545+0000 7f958effd640 1 -- 172.21.15.114:0/2613490337 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f957c004eb0 con 0x7f959813e1e0 2024-03-29T11:23:19.546 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.545+0000 7f958effd640 1 -- 172.21.15.114:0/2613490337 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1016835801 0 0) 0x7f9598140e40 con 0x7f9598071c30 2024-03-29T11:23:19.546 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.545+0000 7f958effd640 10 monclient(hunting): _negotiate 2024-03-29T11:23:19.546 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.545+0000 7f958effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:19.546 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.545+0000 7f958effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:19.546 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.545+0000 7f958effd640 10 monclient(hunting): my global_id is 4323 2024-03-29T11:23:19.546 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.545+0000 7f958effd640 1 -- 172.21.15.114:0/2613490337 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f959813fa90 con 0x7f9598071c30 2024-03-29T11:23:19.546 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.545+0000 7f958effd640 1 -- 172.21.15.114:0/2613490337 <== mon.2 v1:172.21.15.114:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3548486691 0 0) 0x7f9598142510 con 0x7f95980a9750 2024-03-29T11:23:19.547 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.545+0000 7f958effd640 1 -- 172.21.15.114:0/2613490337 >> v1:172.21.15.176:6789/0 conn(0x7f959813e1e0 legacy=0x7f95981d6f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:19.547 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.545+0000 7f958effd640 1 -- 172.21.15.114:0/2613490337 >> v1:172.21.15.114:6789/0 conn(0x7f9598071c30 legacy=0x7f95981404e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:19.547 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.545+0000 7f958effd640 10 monclient: _finish_hunting 0 2024-03-29T11:23:19.547 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.545+0000 7f958effd640 1 monclient: found mon.c 2024-03-29T11:23:19.547 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.545+0000 7f958effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:19.547 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.545+0000 7f958effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:23:19.547 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.545+0000 7f958effd640 1 -- 172.21.15.114:0/2613490337 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95981426a0 con 0x7f95980a9750 2024-03-29T11:23:19.547 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.545+0000 7f958effd640 10 monclient: _finish_auth 0 2024-03-29T11:23:19.547 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.545+0000 7f958effd640 10 monclient: _check_auth_tickets 2024-03-29T11:23:19.547 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.545+0000 7f958effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:19.547 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.545+0000 7f95a0625640 5 monclient: authenticate success, global_id 4316 2024-03-29T11:23:19.547 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.545+0000 7f95a0625640 10 monclient: _renew_subs 2024-03-29T11:23:19.547 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.545+0000 7f95a0625640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:23:19.547 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.545+0000 7f95a0625640 1 -- 172.21.15.114:0/2613490337 --> v1:172.21.15.114:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9598142950 con 0x7f95980a9750 2024-03-29T11:23:19.547 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.545+0000 7f95a0625640 10 client.4316.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:23:19.548 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.545+0000 7f95a0625640 10 monclient: _renew_subs 2024-03-29T11:23:19.548 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.545+0000 7f95a0625640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:23:19.548 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.545+0000 7f95a0625640 1 -- 172.21.15.114:0/2613490337 --> v1:172.21.15.114:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f95981dee50 con 0x7f95980a9750 2024-03-29T11:23:19.548 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.545+0000 7f95a0625640 1 librados: init done 2024-03-29T11:23:19.548 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.546+0000 7f958effd640 1 -- 172.21.15.114:0/2613490337 <== mon.2 v1:172.21.15.114:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9588001930 con 0x7f95980a9750 2024-03-29T11:23:19.548 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.546+0000 7f958effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:19.548 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.546+0000 7f958effd640 1 -- 172.21.15.114:0/2613490337 <== mon.2 v1:172.21.15.114:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f9588004a00 con 0x7f95980a9750 2024-03-29T11:23:19.548 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.546+0000 7f958effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:19.548 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.546+0000 7f958effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:23:19.548 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.546+0000 7f958effd640 10 monclient: dump: 2024-03-29T11:23:19.548 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:23:19.548 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:23:19.548 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:23:19.548 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:23:19.549 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:19.549 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:23:19.549 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:23:19.549 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:23:19.549 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:23:19.549 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:23:19.549 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.546+0000 7f959cbbe640 4 set_mon_vals no callback set 2024-03-29T11:23:19.549 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.547+0000 7f95a0625640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:19.549 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.547+0000 7f95a0625640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:19.549 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.547+0000 7f95a0625640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:23:19.549 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.547+0000 7f95a0625640 1 -- 172.21.15.114:0/2613490337 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9580005080 con 0x7f95980a9750 2024-03-29T11:23:19.549 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.548+0000 7f958effd640 1 -- 172.21.15.114:0/2613490337 <== mon.2 v1:172.21.15.114:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1594292461 0 0) 0x7f958801a9d0 con 0x7f95980a9750 2024-03-29T11:23:19.549 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.549+0000 7f958effd640 1 -- 172.21.15.114:0/2613490337 <== mon.2 v1:172.21.15.114:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3238987313 0 0) 0x7f9588084990 con 0x7f95980a9750 2024-03-29T11:23:19.549 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.549+0000 7f959e3c1640 10 monclient: get_auth_request con 0x7f957c0765f0 auth_method 0 2024-03-29T11:23:19.550 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.549+0000 7f958effd640 10 client.4316.objecter ms_dispatch 0x7f9598000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:23:19.550 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.549+0000 7f958effd640 3 client.4316.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:23:19.550 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.549+0000 7f958effd640 3 client.4316.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:23:19.550 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.549+0000 7f958effd640 20 client.4316.objecter dump_active .. 0 homeless 2024-03-29T11:23:19.550 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.549+0000 7f958effd640 10 client.4316.objecter ms_handle_connect 0x7f957c0765f0 2024-03-29T11:23:19.550 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.556+0000 7f958effd640 1 -- 172.21.15.114:0/2613490337 <== mon.2 v1:172.21.15.114:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9588050270 con 0x7f95980a9750 2024-03-29T11:23:19.550 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.556+0000 7f958effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:19.550 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.556+0000 7f958effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:23:19.849 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.869+0000 7f95a0625640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:19.849 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.869+0000 7f95a0625640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:19.849 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.869+0000 7f95a0625640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:23:19.849 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.869+0000 7f95a0625640 1 -- 172.21.15.114:0/2613490337 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9580005330 con 0x7f95980a9750 2024-03-29T11:23:19.849 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.870+0000 7f958effd640 1 -- 172.21.15.114:0/2613490337 <== mon.2 v1:172.21.15.114:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2858224613) 0x7f95880521a0 con 0x7f95980a9750 2024-03-29T11:23:19.849 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.870+0000 7f958effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:19.849 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.870+0000 7f958effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:23:19.850 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T11:23:19.850 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"b0de2b9b-c45a-44c7-b6ee-e83b39a1e108","created":"2024-03-29T11:20:49.420660+0000","modified":"2024-03-29T11:20:51.700167+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:51.700167+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a64785d3-6b11-499a-b706-cc827ab96a74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5882dadb-e7a8-4604-ba9a-418013c106c8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d7e8c409-bc66-4e4a-bbdb-420bad31388e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0682989-3595-4cd6-90a5-ab9111c278e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"67068120-e3ff-4817-bd06-44feb7479a35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09e8b-8a85-4997-bdea-75d754e6ee67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2f0d9ead-f47f-4cb3-92a4-0a5ad23cd831","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fdb6a25f-6bd8-4c65-9b9b-b460ba7673a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:23:19.852 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.873+0000 7f95a0625640 10 librados: watch_flush enter 2024-03-29T11:23:19.852 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.873+0000 7f95a0625640 10 librados: watch_flush exit 2024-03-29T11:23:19.852 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.873+0000 7f95a0625640 20 client.4316.objecter shutdown clearing up homeless session... 2024-03-29T11:23:19.852 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.873+0000 7f95a0625640 10 client.4316.objecter successfully canceled tick 2024-03-29T11:23:19.852 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.873+0000 7f95a0625640 1 -- 172.21.15.114:0/2613490337 >> v1:172.21.15.114:6816/4038755055 conn(0x7f957c0765f0 legacy=0x7f957c078ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:19.852 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.873+0000 7f95a0625640 10 monclient: shutdown 2024-03-29T11:23:19.852 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.873+0000 7f95a0625640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:19.852 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.873+0000 7f95a0625640 1 -- 172.21.15.114:0/2613490337 >> v1:172.21.15.114:6790/0 conn(0x7f95980a9750 legacy=0x7f95981d6820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:19.852 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.873+0000 7f95a0625640 1 -- 172.21.15.114:0/2613490337 shutdown_connections 2024-03-29T11:23:19.853 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.873+0000 7f95a0625640 1 -- 172.21.15.114:0/2613490337 >> 172.21.15.114:0/2613490337 conn(0x7f959806b660 msgr2=0x7f959813aae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:19.853 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.874+0000 7f95a0625640 1 -- 172.21.15.114:0/2613490337 shutdown_connections 2024-03-29T11:23:19.853 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.874+0000 7f95a0625640 1 -- 172.21.15.114:0/2613490337 wait complete. 2024-03-29T11:23:19.853 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:19.874+0000 7f95a0625640 1 librados: shutdown 2024-03-29T11:23:19.868 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:23:26.674 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:23:26.822 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.843+0000 7f91e0953640 10 monclient: get_monmap_and_config 2024-03-29T11:23:26.823 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.843+0000 7f91e0953640 10 monclient: build_initial_monmap 2024-03-29T11:23:26.823 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.843+0000 7f91e0953640 10 monclient: monmap: 2024-03-29T11:23:26.823 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:23:26.823 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:23:26.823 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:23:26.844122+0000 2024-03-29T11:23:26.823 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:23:26.844122+0000 2024-03-29T11:23:26.823 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:26.823 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:23:26.823 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:23:26.823 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T11:23:26.823 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T11:23:26.823 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:23:26.824 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.844+0000 7f91e0953640 1 Processor -- start 2024-03-29T11:23:26.825 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.844+0000 7f91e0953640 1 -- start start 2024-03-29T11:23:26.825 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.844+0000 7f91e0953640 10 monclient: init 2024-03-29T11:23:26.825 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.845+0000 7f91e0953640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:26.825 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.845+0000 7f91e0953640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:23:26.825 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.845+0000 7f91e0953640 10 monclient(hunting): picked mon.noname-b con 0x7f91d4006800 addr v1:172.21.15.114:6790/0 2024-03-29T11:23:26.825 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.845+0000 7f91e0953640 10 monclient(hunting): picked mon.noname-a con 0x7f91d4006be0 addr v1:172.21.15.114:6789/0 2024-03-29T11:23:26.825 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.845+0000 7f91e0953640 10 monclient(hunting): picked mon.noname-c con 0x7f91d4031b60 addr v1:172.21.15.176:6789/0 2024-03-29T11:23:26.825 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.845+0000 7f91e0953640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91d4029370 con 0x7f91d4006be0 2024-03-29T11:23:26.825 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.845+0000 7f91e0953640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91d40295e0 con 0x7f91d4031b60 2024-03-29T11:23:26.825 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.845+0000 7f91e0953640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91d4031360 con 0x7f91d4006800 2024-03-29T11:23:26.825 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.845+0000 7f91e0953640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:26.825 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.845+0000 7f91e0953640 10 monclient(hunting): authenticate will time out at 1077.027802s 2024-03-29T11:23:26.825 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.846+0000 7f91db7fe640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f91d4006800 0x7f91d40cad20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:58724/0 (socket says 172.21.15.114:58724) 2024-03-29T11:23:26.826 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.846+0000 7f91db7fe640 1 -- 172.21.15.114:0/1470780759 learned_addr learned my addr 172.21.15.114:0/1470780759 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:23:26.826 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.846+0000 7f91da7fc640 1 -- 172.21.15.114:0/1470780759 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 895398727 0 0) 0x7f91d4031360 con 0x7f91d4006800 2024-03-29T11:23:26.826 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.846+0000 7f91da7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:23:26.826 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.846+0000 7f91da7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:26.826 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.846+0000 7f91da7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:26.826 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.846+0000 7f91da7fc640 10 monclient(hunting): my global_id is 4319 2024-03-29T11:23:26.826 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.846+0000 7f91da7fc640 1 -- 172.21.15.114:0/1470780759 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91cc003520 con 0x7f91d4006800 2024-03-29T11:23:26.827 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.846+0000 7f91da7fc640 1 -- 172.21.15.114:0/1470780759 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1578688468 0 0) 0x7f91cc003520 con 0x7f91d4006800 2024-03-29T11:23:26.827 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.847+0000 7f91da7fc640 1 -- 172.21.15.114:0/1470780759 >> v1:172.21.15.176:6789/0 conn(0x7f91d4031b60 legacy=0x7f91d40d30c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:23:26.827 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.847+0000 7f91da7fc640 1 -- 172.21.15.114:0/1470780759 >> v1:172.21.15.114:6789/0 conn(0x7f91d4006be0 legacy=0x7f91d40c4ad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:23:26.828 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.849+0000 7f91da7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:23:26.828 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.849+0000 7f91da7fc640 1 monclient: found mon.noname-b 2024-03-29T11:23:26.828 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.849+0000 7f91da7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:26.829 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.849+0000 7f91da7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.114:6790/0 2024-03-29T11:23:26.829 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.849+0000 7f91da7fc640 1 -- 172.21.15.114:0/1470780759 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91d4031640 con 0x7f91d4006800 2024-03-29T11:23:26.829 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.849+0000 7f91da7fc640 10 monclient: _finish_auth 0 2024-03-29T11:23:26.829 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.849+0000 7f91da7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:23:26.829 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.849+0000 7f91da7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:26.829 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.849+0000 7f91da7fc640 1 -- 172.21.15.114:0/1470780759 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f91c8001690 con 0x7f91d4006800 2024-03-29T11:23:26.829 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.849+0000 7f91da7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:26.829 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.849+0000 7f91da7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:23:26.829 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.849+0000 7f91da7fc640 10 monclient: dump: 2024-03-29T11:23:26.829 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:23:26.829 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:23:26.829 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:23:26.829 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:23:26.829 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:26.829 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:23:26.829 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:23:26.830 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:23:26.830 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:23:26.830 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:23:26.832 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.849+0000 7f91e0953640 5 monclient: authenticate success, global_id 4319 2024-03-29T11:23:26.832 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.849+0000 7f91e0953640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:26.832 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.852+0000 7f91da7fc640 1 -- 172.21.15.114:0/1470780759 <== mon.2 v1:172.21.15.114:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f91c80018b0 con 0x7f91d4006800 2024-03-29T11:23:26.832 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.852+0000 7f91da7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:26.832 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.852+0000 7f91da7fc640 1 -- 172.21.15.114:0/1470780759 <== mon.2 v1:172.21.15.114:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f91c8001c40 con 0x7f91d4006800 2024-03-29T11:23:26.832 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.852+0000 7f91da7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:26.832 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.852+0000 7f91da7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:23:26.832 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.852+0000 7f91da7fc640 10 monclient: dump: 2024-03-29T11:23:26.832 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:23:26.832 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:23:26.832 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:23:26.832 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:23:26.833 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:26.833 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:23:26.833 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:23:26.833 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:23:26.833 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:23:26.833 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:23:26.834 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.853+0000 7f91e0953640 10 monclient: get_monmap_and_config success 2024-03-29T11:23:26.834 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.853+0000 7f91e0953640 4 set_mon_vals no callback set 2024-03-29T11:23:26.834 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.854+0000 7f91e0953640 10 monclient: shutdown 2024-03-29T11:23:26.834 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.854+0000 7f91e0953640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:26.834 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.854+0000 7f91e0953640 1 -- 172.21.15.114:0/1470780759 >> v1:172.21.15.114:6790/0 conn(0x7f91d4006800 legacy=0x7f91d40cad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:26.834 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.854+0000 7f91e0953640 1 -- 172.21.15.114:0/1470780759 shutdown_connections 2024-03-29T11:23:26.834 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.854+0000 7f91e0953640 1 -- 172.21.15.114:0/1470780759 >> 172.21.15.114:0/1470780759 conn(0x7f91d40058f0 msgr2=0x7f91d4005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:26.834 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.855+0000 7f91e0953640 1 -- 172.21.15.114:0/1470780759 shutdown_connections 2024-03-29T11:23:26.835 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.855+0000 7f91e0953640 1 -- 172.21.15.114:0/1470780759 wait complete. 2024-03-29T11:23:26.835 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.856+0000 7f91e0953640 10 monclient: build_initial_monmap 2024-03-29T11:23:26.835 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.856+0000 7f91e0953640 10 monclient: monmap: 2024-03-29T11:23:26.835 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:23:26.835 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:23:26.835 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T11:23:26.835 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T11:23:26.835 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:26.836 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:23:26.836 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:23:26.836 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T11:23:26.836 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T11:23:26.836 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:23:26.837 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.857+0000 7f91e0953640 1 librados: starting msgr at 2024-03-29T11:23:26.837 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.857+0000 7f91e0953640 1 librados: starting objecter 2024-03-29T11:23:26.837 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.857+0000 7f91e0953640 1 Processor -- start 2024-03-29T11:23:26.837 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.857+0000 7f91e0953640 1 -- start start 2024-03-29T11:23:26.837 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.857+0000 7f91e0953640 1 librados: setting wanted keys 2024-03-29T11:23:26.837 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.857+0000 7f91e0953640 1 librados: calling monclient init 2024-03-29T11:23:26.837 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.857+0000 7f91e0953640 10 monclient: init 2024-03-29T11:23:26.840 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.858+0000 7f91e0953640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:26.840 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.858+0000 7f91e0953640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:23:26.840 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.858+0000 7f91e0953640 10 monclient(hunting): picked mon.noname-a con 0x7f91d4006800 addr v1:172.21.15.114:6789/0 2024-03-29T11:23:26.840 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.858+0000 7f91e0953640 10 monclient(hunting): picked mon.noname-b con 0x7f91d4031b60 addr v1:172.21.15.176:6789/0 2024-03-29T11:23:26.840 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.858+0000 7f91e0953640 10 monclient(hunting): picked mon.noname-c con 0x7f91d40c11a0 addr v1:172.21.15.114:6790/0 2024-03-29T11:23:26.840 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.858+0000 7f91e0953640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91d40a2280 con 0x7f91d4006800 2024-03-29T11:23:26.840 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.858+0000 7f91e0953640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91d40a3f60 con 0x7f91d4031b60 2024-03-29T11:23:26.840 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.858+0000 7f91e0953640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91d40a2b00 con 0x7f91d40c11a0 2024-03-29T11:23:26.840 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.858+0000 7f91e0953640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:26.840 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.859+0000 7f91e0953640 10 monclient(hunting): authenticate will time out at 1077.041101s 2024-03-29T11:23:26.840 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.859+0000 7f91dbfff640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f91d40c11a0 0x7f91d415f910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:34290/0 (socket says 172.21.15.114:34290) 2024-03-29T11:23:26.841 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.859+0000 7f91dbfff640 1 -- 172.21.15.114:0/3755559319 learned_addr learned my addr 172.21.15.114:0/3755559319 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:23:26.841 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.859+0000 7f91c7fff640 10 client.?.objecter ms_handle_connect 0x7f91d40c11a0 2024-03-29T11:23:26.841 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.859+0000 7f91c7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:26.841 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.859+0000 7f91c7fff640 1 -- 172.21.15.114:0/3755559319 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3397967240 0 0) 0x7f91d40a2b00 con 0x7f91d40c11a0 2024-03-29T11:23:26.841 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.859+0000 7f91c7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:23:26.841 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.859+0000 7f91c7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:26.841 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.859+0000 7f91c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:26.841 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.859+0000 7f91c7fff640 10 monclient(hunting): my global_id is 4322 2024-03-29T11:23:26.841 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.859+0000 7f91c7fff640 1 -- 172.21.15.114:0/3755559319 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91c0004eb0 con 0x7f91d40c11a0 2024-03-29T11:23:26.841 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.860+0000 7f91c7fff640 1 -- 172.21.15.114:0/3755559319 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4192561872 0 0) 0x7f91c0004eb0 con 0x7f91d40c11a0 2024-03-29T11:23:26.841 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.860+0000 7f91c7fff640 1 -- 172.21.15.114:0/3755559319 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f91d40a2b00 con 0x7f91d40c11a0 2024-03-29T11:23:26.841 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.860+0000 7f91c7fff640 1 -- 172.21.15.114:0/3755559319 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f91dc04fdd0 con 0x7f91d40c11a0 2024-03-29T11:23:26.841 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.860+0000 7f91c7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:26.841 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.860+0000 7f91c7fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:23:26.841 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.860+0000 7f91c7fff640 10 monclient(hunting): dump: 2024-03-29T11:23:26.842 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:23:26.842 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:23:26.842 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:23:26.842 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:23:26.842 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:26.842 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:23:26.842 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:23:26.842 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:23:26.842 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:23:26.842 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:23:26.842 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.860+0000 7f91c7fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:23:26.845 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.862+0000 7f91c7fff640 10 client.?.objecter ms_handle_connect 0x7f91d4031b60 2024-03-29T11:23:26.846 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.862+0000 7f91c7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:26.846 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.862+0000 7f91c7fff640 1 -- 172.21.15.114:0/3755559319 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3701987815 0 0) 0x7f91d40a3f60 con 0x7f91d4031b60 2024-03-29T11:23:26.846 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.862+0000 7f91c7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:23:26.846 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.862+0000 7f91c7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:26.846 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.862+0000 7f91c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:26.846 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.862+0000 7f91c7fff640 10 monclient(hunting): my global_id is 4321 2024-03-29T11:23:26.846 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.862+0000 7f91c7fff640 1 -- 172.21.15.114:0/3755559319 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91c0004eb0 con 0x7f91d4031b60 2024-03-29T11:23:26.846 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.863+0000 7f91c7fff640 1 -- 172.21.15.114:0/3755559319 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1248686173 0 0) 0x7f91c0004eb0 con 0x7f91d4031b60 2024-03-29T11:23:26.846 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.863+0000 7f91c7fff640 1 -- 172.21.15.114:0/3755559319 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f91d40a3f60 con 0x7f91d4031b60 2024-03-29T11:23:26.846 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.863+0000 7f91c7fff640 1 -- 172.21.15.114:0/3755559319 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f91d0001850 con 0x7f91d4031b60 2024-03-29T11:23:26.846 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.863+0000 7f91c7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:26.846 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.863+0000 7f91c7fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:23:26.846 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.863+0000 7f91c7fff640 10 monclient(hunting): dump: 2024-03-29T11:23:26.847 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:23:26.847 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:23:26.847 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:23:26.847 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:23:26.847 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:26.847 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:23:26.847 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:23:26.847 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:23:26.847 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:23:26.847 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:23:26.847 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.863+0000 7f91c7fff640 1 -- 172.21.15.114:0/3755559319 <== mon.2 v1:172.21.15.114:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2164385166 0 0) 0x7f91d40a2b00 con 0x7f91d40c11a0 2024-03-29T11:23:26.847 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.863+0000 7f91c7fff640 1 -- 172.21.15.114:0/3755559319 >> v1:172.21.15.176:6789/0 conn(0x7f91d4031b60 legacy=0x7f91d40c0a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:26.847 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.863+0000 7f91c7fff640 1 -- 172.21.15.114:0/3755559319 >> v1:172.21.15.114:6789/0 conn(0x7f91d4006800 legacy=0x7f91d40a1b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:23:26.847 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.865+0000 7f91c7fff640 10 monclient: _finish_hunting 0 2024-03-29T11:23:26.847 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.865+0000 7f91c7fff640 1 monclient: found mon.c 2024-03-29T11:23:26.848 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.865+0000 7f91c7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:26.848 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.865+0000 7f91c7fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:23:26.848 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.865+0000 7f91c7fff640 1 -- 172.21.15.114:0/3755559319 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91d40a2430 con 0x7f91d40c11a0 2024-03-29T11:23:26.848 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.865+0000 7f91c7fff640 10 monclient: _finish_auth 0 2024-03-29T11:23:26.848 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.865+0000 7f91c7fff640 10 monclient: _check_auth_tickets 2024-03-29T11:23:26.848 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.865+0000 7f91c7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:26.848 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.865+0000 7f91e0953640 5 monclient: authenticate success, global_id 4322 2024-03-29T11:23:26.848 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.865+0000 7f91e0953640 10 monclient: _renew_subs 2024-03-29T11:23:26.848 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.865+0000 7f91e0953640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:23:26.848 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.865+0000 7f91e0953640 1 -- 172.21.15.114:0/3755559319 --> v1:172.21.15.114:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91d40a26e0 con 0x7f91d40c11a0 2024-03-29T11:23:26.848 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.865+0000 7f91e0953640 10 client.4322.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:23:26.848 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.865+0000 7f91e0953640 10 monclient: _renew_subs 2024-03-29T11:23:26.848 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.865+0000 7f91e0953640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:23:26.849 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.865+0000 7f91e0953640 1 -- 172.21.15.114:0/3755559319 --> v1:172.21.15.114:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f91d4160200 con 0x7f91d40c11a0 2024-03-29T11:23:26.849 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.865+0000 7f91e0953640 1 librados: init done 2024-03-29T11:23:26.850 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.868+0000 7f91c7fff640 1 -- 172.21.15.114:0/3755559319 <== mon.2 v1:172.21.15.114:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f91dc04f400 con 0x7f91d40c11a0 2024-03-29T11:23:26.850 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.868+0000 7f91c7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:26.850 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.868+0000 7f91c7fff640 1 -- 172.21.15.114:0/3755559319 <== mon.2 v1:172.21.15.114:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f91dc075f10 con 0x7f91d40c11a0 2024-03-29T11:23:26.851 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.868+0000 7f91c7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:26.851 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.868+0000 7f91c7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:23:26.851 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.868+0000 7f91c7fff640 10 monclient: dump: 2024-03-29T11:23:26.851 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:23:26.851 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:23:26.851 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:23:26.851 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:23:26.851 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:26.851 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:23:26.851 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:23:26.851 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:23:26.851 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:23:26.851 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:23:26.851 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.868+0000 7f91c7fff640 1 -- 172.21.15.114:0/3755559319 <== mon.2 v1:172.21.15.114:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1594292461 0 0) 0x7f91dc067bf0 con 0x7f91d40c11a0 2024-03-29T11:23:26.851 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.869+0000 7f91c7fff640 1 -- 172.21.15.114:0/3755559319 <== mon.2 v1:172.21.15.114:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3238987313 0 0) 0x7f91dc0ff530 con 0x7f91d40c11a0 2024-03-29T11:23:26.852 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.869+0000 7f91c7fff640 10 client.4322.objecter ms_dispatch 0x7f91dc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:23:26.852 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.869+0000 7f91c7fff640 3 client.4322.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:23:26.852 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.869+0000 7f91c7fff640 3 client.4322.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:23:26.852 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.869+0000 7f91c7fff640 20 client.4322.objecter dump_active .. 0 homeless 2024-03-29T11:23:26.852 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.869+0000 7f91d97fa640 4 set_mon_vals no callback set 2024-03-29T11:23:26.852 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.869+0000 7f91db7fe640 10 monclient: get_auth_request con 0x7f91c006dca0 auth_method 0 2024-03-29T11:23:26.852 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.870+0000 7f91c5ffb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:26.852 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.870+0000 7f91c5ffb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:26.852 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.870+0000 7f91c5ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:23:26.852 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.870+0000 7f91c5ffb640 1 -- 172.21.15.114:0/3755559319 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91d4004cd0 con 0x7f91d40c11a0 2024-03-29T11:23:26.852 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.870+0000 7f91c7fff640 10 client.4322.objecter ms_handle_connect 0x7f91c006dca0 2024-03-29T11:23:26.857 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.877+0000 7f91c7fff640 1 -- 172.21.15.114:0/3755559319 <== mon.2 v1:172.21.15.114:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f91dc0cae90 con 0x7f91d40c11a0 2024-03-29T11:23:26.857 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.877+0000 7f91c7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:26.858 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:26.877+0000 7f91c7fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:23:27.173 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:27.194+0000 7f91c5ffb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:27.173 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:27.194+0000 7f91c5ffb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:27.173 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:27.194+0000 7f91c5ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:23:27.174 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:27.194+0000 7f91c5ffb640 1 -- 172.21.15.114:0/3755559319 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f91d40a2c90 con 0x7f91d40c11a0 2024-03-29T11:23:27.174 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:27.195+0000 7f91c7fff640 1 -- 172.21.15.114:0/3755559319 <== mon.2 v1:172.21.15.114:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2858224613) 0x7f91dc0ccdc0 con 0x7f91d40c11a0 2024-03-29T11:23:27.175 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:27.195+0000 7f91c7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:27.175 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:27.195+0000 7f91c7fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:23:27.175 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T11:23:27.175 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"b0de2b9b-c45a-44c7-b6ee-e83b39a1e108","created":"2024-03-29T11:20:49.420660+0000","modified":"2024-03-29T11:20:51.700167+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:51.700167+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a64785d3-6b11-499a-b706-cc827ab96a74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5882dadb-e7a8-4604-ba9a-418013c106c8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d7e8c409-bc66-4e4a-bbdb-420bad31388e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0682989-3595-4cd6-90a5-ab9111c278e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"67068120-e3ff-4817-bd06-44feb7479a35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09e8b-8a85-4997-bdea-75d754e6ee67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2f0d9ead-f47f-4cb3-92a4-0a5ad23cd831","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fdb6a25f-6bd8-4c65-9b9b-b460ba7673a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:23:27.178 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:27.199+0000 7f91c5ffb640 10 librados: watch_flush enter 2024-03-29T11:23:27.178 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:27.199+0000 7f91c5ffb640 10 librados: watch_flush exit 2024-03-29T11:23:27.178 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:27.199+0000 7f91c5ffb640 20 client.4322.objecter shutdown clearing up homeless session... 2024-03-29T11:23:27.178 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:27.199+0000 7f91c5ffb640 10 client.4322.objecter successfully canceled tick 2024-03-29T11:23:27.179 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:27.200+0000 7f91c5ffb640 1 -- 172.21.15.114:0/3755559319 >> v1:172.21.15.114:6816/4038755055 conn(0x7f91c006dca0 legacy=0x7f91c0070160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:27.179 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:27.200+0000 7f91c5ffb640 10 monclient: shutdown 2024-03-29T11:23:27.179 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:27.200+0000 7f91c5ffb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:27.179 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:27.200+0000 7f91c5ffb640 1 -- 172.21.15.114:0/3755559319 >> v1:172.21.15.114:6790/0 conn(0x7f91d40c11a0 legacy=0x7f91d415f910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:27.180 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:27.201+0000 7f91c5ffb640 1 -- 172.21.15.114:0/3755559319 shutdown_connections 2024-03-29T11:23:27.180 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:27.201+0000 7f91c5ffb640 1 -- 172.21.15.114:0/3755559319 >> 172.21.15.114:0/3755559319 conn(0x7f91d40058f0 msgr2=0x7f91d402a430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:27.182 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:27.203+0000 7f91c5ffb640 1 -- 172.21.15.114:0/3755559319 shutdown_connections 2024-03-29T11:23:27.182 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:27.203+0000 7f91c5ffb640 1 -- 172.21.15.114:0/3755559319 wait complete. 2024-03-29T11:23:27.182 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:27.203+0000 7f91c5ffb640 1 librados: shutdown 2024-03-29T11:23:27.198 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:23:34.005 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:23:34.155 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.175+0000 7fcc309ec640 10 monclient: get_monmap_and_config 2024-03-29T11:23:34.155 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.175+0000 7fcc309ec640 10 monclient: build_initial_monmap 2024-03-29T11:23:34.155 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.175+0000 7fcc309ec640 10 monclient: monmap: 2024-03-29T11:23:34.156 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:23:34.156 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:23:34.156 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:23:34.176939+0000 2024-03-29T11:23:34.156 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:23:34.176939+0000 2024-03-29T11:23:34.156 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:34.156 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:23:34.156 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:23:34.156 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T11:23:34.156 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T11:23:34.156 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:23:34.157 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.177+0000 7fcc309ec640 1 Processor -- start 2024-03-29T11:23:34.157 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.177+0000 7fcc309ec640 1 -- start start 2024-03-29T11:23:34.160 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.177+0000 7fcc309ec640 10 monclient: init 2024-03-29T11:23:34.160 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.178+0000 7fcc309ec640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:34.160 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.178+0000 7fcc309ec640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:23:34.160 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.178+0000 7fcc309ec640 10 monclient(hunting): picked mon.noname-b con 0x7fcc2402b4c0 addr v1:172.21.15.114:6790/0 2024-03-29T11:23:34.160 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.178+0000 7fcc309ec640 10 monclient(hunting): picked mon.noname-a con 0x7fcc2400b8a0 addr v1:172.21.15.114:6789/0 2024-03-29T11:23:34.160 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.178+0000 7fcc309ec640 10 monclient(hunting): picked mon.noname-c con 0x7fcc2402a5a0 addr v1:172.21.15.176:6789/0 2024-03-29T11:23:34.160 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.178+0000 7fcc309ec640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc2402c5c0 con 0x7fcc2400b8a0 2024-03-29T11:23:34.160 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.178+0000 7fcc309ec640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc240c11a0 con 0x7fcc2402a5a0 2024-03-29T11:23:34.160 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.178+0000 7fcc309ec640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc2402c830 con 0x7fcc2402b4c0 2024-03-29T11:23:34.160 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.178+0000 7fcc309ec640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:34.161 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.178+0000 7fcc309ec640 10 monclient(hunting): authenticate will time out at 1084.360418s 2024-03-29T11:23:34.162 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.182+0000 7fcc2affd640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fcc2400b8a0 0x7fcc240c0a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:36705/0 (socket says 172.21.15.114:36705) 2024-03-29T11:23:34.162 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.182+0000 7fcc2affd640 1 -- 172.21.15.114:0/2747823398 learned_addr learned my addr 172.21.15.114:0/2747823398 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:23:34.162 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.183+0000 7fcc2a7fc640 1 -- 172.21.15.114:0/2747823398 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 149925447 0 0) 0x7fcc2402c5c0 con 0x7fcc2400b8a0 2024-03-29T11:23:34.162 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.183+0000 7fcc2a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:23:34.163 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.183+0000 7fcc2a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:34.163 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.183+0000 7fcc2a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:34.163 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.183+0000 7fcc2a7fc640 10 monclient(hunting): my global_id is 4326 2024-03-29T11:23:34.163 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.183+0000 7fcc2a7fc640 1 -- 172.21.15.114:0/2747823398 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc1c003520 con 0x7fcc2400b8a0 2024-03-29T11:23:34.164 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.185+0000 7fcc2a7fc640 1 -- 172.21.15.114:0/2747823398 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1808560152 0 0) 0x7fcc1c003520 con 0x7fcc2400b8a0 2024-03-29T11:23:34.164 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.185+0000 7fcc2a7fc640 1 -- 172.21.15.114:0/2747823398 >> v1:172.21.15.114:6790/0 conn(0x7fcc2402b4c0 legacy=0x7fcc24006890 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:23:34.165 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.185+0000 7fcc2a7fc640 1 -- 172.21.15.114:0/2747823398 >> v1:172.21.15.176:6789/0 conn(0x7fcc2402a5a0 legacy=0x7fcc240c6d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:34.165 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.186+0000 7fcc2a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:23:34.166 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.186+0000 7fcc2a7fc640 1 monclient: found mon.noname-a 2024-03-29T11:23:34.166 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.186+0000 7fcc2a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:34.166 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.186+0000 7fcc2a7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.114:6789/0 2024-03-29T11:23:34.166 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.186+0000 7fcc2a7fc640 1 -- 172.21.15.114:0/2747823398 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc2402afa0 con 0x7fcc2400b8a0 2024-03-29T11:23:34.166 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.186+0000 7fcc2a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:23:34.166 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.186+0000 7fcc2a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:23:34.166 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.186+0000 7fcc2a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:34.166 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.186+0000 7fcc309ec640 5 monclient: authenticate success, global_id 4326 2024-03-29T11:23:34.166 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.186+0000 7fcc309ec640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:34.166 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.186+0000 7fcc2a7fc640 1 -- 172.21.15.114:0/2747823398 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fcc18001690 con 0x7fcc2400b8a0 2024-03-29T11:23:34.166 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.186+0000 7fcc2a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:34.166 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.186+0000 7fcc2a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:23:34.166 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.186+0000 7fcc2a7fc640 10 monclient: dump: 2024-03-29T11:23:34.167 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:23:34.167 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:23:34.167 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:23:34.167 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:23:34.167 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:34.167 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:23:34.167 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:23:34.167 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:23:34.167 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:23:34.167 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:23:34.167 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.186+0000 7fcc309ec640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:34.168 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.186+0000 7fcc2a7fc640 1 -- 172.21.15.114:0/2747823398 <== mon.0 v1:172.21.15.114:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcc180017f0 con 0x7fcc2400b8a0 2024-03-29T11:23:34.168 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.186+0000 7fcc2a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:34.168 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.186+0000 7fcc309ec640 10 monclient: get_monmap_and_config success 2024-03-29T11:23:34.168 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.186+0000 7fcc309ec640 4 set_mon_vals no callback set 2024-03-29T11:23:34.168 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.187+0000 7fcc2a7fc640 1 -- 172.21.15.114:0/2747823398 <== mon.0 v1:172.21.15.114:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fcc18001c00 con 0x7fcc2400b8a0 2024-03-29T11:23:34.168 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.187+0000 7fcc2a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:34.168 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.187+0000 7fcc2a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:23:34.169 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.187+0000 7fcc2a7fc640 10 monclient: dump: 2024-03-29T11:23:34.169 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:23:34.169 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:23:34.169 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:23:34.169 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:23:34.169 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:34.169 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:23:34.169 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:23:34.169 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:23:34.169 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:23:34.169 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:23:34.169 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.187+0000 7fcc309ec640 10 monclient: shutdown 2024-03-29T11:23:34.169 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.187+0000 7fcc309ec640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:34.169 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.187+0000 7fcc309ec640 1 -- 172.21.15.114:0/2747823398 >> v1:172.21.15.114:6789/0 conn(0x7fcc2400b8a0 legacy=0x7fcc240c0a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:34.170 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.187+0000 7fcc309ec640 1 -- 172.21.15.114:0/2747823398 shutdown_connections 2024-03-29T11:23:34.170 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.187+0000 7fcc309ec640 1 -- 172.21.15.114:0/2747823398 >> 172.21.15.114:0/2747823398 conn(0x7fcc240058f0 msgr2=0x7fcc24005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:34.170 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.190+0000 7fcc309ec640 1 -- 172.21.15.114:0/2747823398 shutdown_connections 2024-03-29T11:23:34.170 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.190+0000 7fcc309ec640 1 -- 172.21.15.114:0/2747823398 wait complete. 2024-03-29T11:23:34.174 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.191+0000 7fcc309ec640 10 monclient: build_initial_monmap 2024-03-29T11:23:34.174 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.191+0000 7fcc309ec640 10 monclient: monmap: 2024-03-29T11:23:34.174 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:23:34.174 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:23:34.174 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T11:23:34.174 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T11:23:34.174 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:34.175 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:23:34.175 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:23:34.175 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T11:23:34.175 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T11:23:34.175 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:23:34.175 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.191+0000 7fcc309ec640 1 librados: starting msgr at 2024-03-29T11:23:34.175 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.191+0000 7fcc309ec640 1 librados: starting objecter 2024-03-29T11:23:34.175 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.192+0000 7fcc309ec640 1 Processor -- start 2024-03-29T11:23:34.175 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.192+0000 7fcc309ec640 1 -- start start 2024-03-29T11:23:34.175 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.192+0000 7fcc309ec640 1 librados: setting wanted keys 2024-03-29T11:23:34.175 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.192+0000 7fcc309ec640 1 librados: calling monclient init 2024-03-29T11:23:34.175 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.192+0000 7fcc309ec640 10 monclient: init 2024-03-29T11:23:34.175 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.193+0000 7fcc309ec640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:34.175 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.193+0000 7fcc309ec640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:23:34.175 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.193+0000 7fcc309ec640 10 monclient(hunting): picked mon.noname-a con 0x7fcc2400b8a0 addr v1:172.21.15.114:6789/0 2024-03-29T11:23:34.176 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.193+0000 7fcc309ec640 10 monclient(hunting): picked mon.noname-c con 0x7fcc2402a5a0 addr v1:172.21.15.114:6790/0 2024-03-29T11:23:34.176 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.193+0000 7fcc309ec640 10 monclient(hunting): picked mon.noname-b con 0x7fcc2402b4c0 addr v1:172.21.15.176:6789/0 2024-03-29T11:23:34.176 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.193+0000 7fcc309ec640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc240d6d20 con 0x7fcc2400b8a0 2024-03-29T11:23:34.176 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.193+0000 7fcc309ec640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc240d7fd0 con 0x7fcc2402b4c0 2024-03-29T11:23:34.176 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.193+0000 7fcc309ec640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc240d8240 con 0x7fcc2402a5a0 2024-03-29T11:23:34.176 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.193+0000 7fcc309ec640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:34.176 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.193+0000 7fcc309ec640 10 monclient(hunting): authenticate will time out at 1084.375241s 2024-03-29T11:23:34.176 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.196+0000 7fcc2bfff640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7fcc2402b4c0 0x7fcc24174cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.114:60495/0 (socket says 172.21.15.114:60495) 2024-03-29T11:23:34.176 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.196+0000 7fcc2bfff640 1 -- 172.21.15.114:0/3680840789 learned_addr learned my addr 172.21.15.114:0/3680840789 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:23:34.177 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.198+0000 7fcc17fff640 10 client.?.objecter ms_handle_connect 0x7fcc2402b4c0 2024-03-29T11:23:34.178 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.198+0000 7fcc17fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:34.178 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.198+0000 7fcc17fff640 1 -- 172.21.15.114:0/3680840789 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 157196189 0 0) 0x7fcc240d7fd0 con 0x7fcc2402b4c0 2024-03-29T11:23:34.178 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.198+0000 7fcc17fff640 10 monclient(hunting): _negotiate 2024-03-29T11:23:34.178 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.198+0000 7fcc17fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:34.178 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.198+0000 7fcc17fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:34.178 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.198+0000 7fcc17fff640 10 monclient(hunting): my global_id is 4327 2024-03-29T11:23:34.178 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.198+0000 7fcc17fff640 1 -- 172.21.15.114:0/3680840789 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc10004eb0 con 0x7fcc2402b4c0 2024-03-29T11:23:34.178 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.199+0000 7fcc17fff640 10 client.?.objecter ms_handle_connect 0x7fcc2402a5a0 2024-03-29T11:23:34.178 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.199+0000 7fcc17fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:34.179 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.199+0000 7fcc17fff640 1 -- 172.21.15.114:0/3680840789 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3494527969 0 0) 0x7fcc240d8240 con 0x7fcc2402a5a0 2024-03-29T11:23:34.179 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.199+0000 7fcc17fff640 10 monclient(hunting): _negotiate 2024-03-29T11:23:34.179 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.199+0000 7fcc17fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:34.179 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.199+0000 7fcc17fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:34.179 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.199+0000 7fcc17fff640 10 monclient(hunting): my global_id is 4325 2024-03-29T11:23:34.179 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.199+0000 7fcc17fff640 1 -- 172.21.15.114:0/3680840789 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc240d7fd0 con 0x7fcc2402a5a0 2024-03-29T11:23:34.180 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.199+0000 7fcc17fff640 1 -- 172.21.15.114:0/3680840789 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2369915537 0 0) 0x7fcc240d7fd0 con 0x7fcc2402a5a0 2024-03-29T11:23:34.180 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.199+0000 7fcc17fff640 1 -- 172.21.15.114:0/3680840789 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcc240d8240 con 0x7fcc2402a5a0 2024-03-29T11:23:34.180 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.199+0000 7fcc17fff640 1 -- 172.21.15.114:0/3680840789 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fcc18001bc0 con 0x7fcc2402a5a0 2024-03-29T11:23:34.180 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.199+0000 7fcc17fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:34.180 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.199+0000 7fcc17fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:23:34.180 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.199+0000 7fcc17fff640 10 monclient(hunting): dump: 2024-03-29T11:23:34.180 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:23:34.180 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:23:34.181 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:23:34.181 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:23:34.181 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:34.181 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:23:34.181 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:23:34.181 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:23:34.181 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:23:34.181 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:23:34.181 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.199+0000 7fcc17fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:23:34.182 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.201+0000 7fcc17fff640 1 -- 172.21.15.114:0/3680840789 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3293757482 0 0) 0x7fcc10004eb0 con 0x7fcc2402b4c0 2024-03-29T11:23:34.182 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.201+0000 7fcc17fff640 1 -- 172.21.15.114:0/3680840789 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcc240d7fd0 con 0x7fcc2402b4c0 2024-03-29T11:23:34.182 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.202+0000 7fcc17fff640 1 -- 172.21.15.114:0/3680840789 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fcc2c04fdd0 con 0x7fcc2402b4c0 2024-03-29T11:23:34.182 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.202+0000 7fcc17fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:34.182 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.202+0000 7fcc17fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:23:34.182 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.202+0000 7fcc17fff640 10 monclient(hunting): dump: 2024-03-29T11:23:34.182 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:23:34.182 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:23:34.182 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:23:34.183 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:23:34.183 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:34.183 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:23:34.183 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:23:34.183 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:23:34.183 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:23:34.183 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:23:34.183 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.203+0000 7fcc17fff640 1 -- 172.21.15.114:0/3680840789 <== mon.2 v1:172.21.15.114:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1162547607 0 0) 0x7fcc240d8240 con 0x7fcc2402a5a0 2024-03-29T11:23:34.184 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.203+0000 7fcc17fff640 1 -- 172.21.15.114:0/3680840789 >> v1:172.21.15.176:6789/0 conn(0x7fcc2402b4c0 legacy=0x7fcc24174cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:34.184 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.203+0000 7fcc17fff640 1 -- 172.21.15.114:0/3680840789 >> v1:172.21.15.114:6789/0 conn(0x7fcc2400b8a0 legacy=0x7fcc240d4420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:34.184 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.203+0000 7fcc17fff640 10 monclient: _finish_hunting 0 2024-03-29T11:23:34.184 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.203+0000 7fcc17fff640 1 monclient: found mon.c 2024-03-29T11:23:34.184 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.203+0000 7fcc17fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:34.184 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.203+0000 7fcc17fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:23:34.184 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.203+0000 7fcc17fff640 1 -- 172.21.15.114:0/3680840789 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc240d6960 con 0x7fcc2402a5a0 2024-03-29T11:23:34.184 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.203+0000 7fcc17fff640 10 monclient: _finish_auth 0 2024-03-29T11:23:34.184 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.203+0000 7fcc17fff640 10 monclient: _check_auth_tickets 2024-03-29T11:23:34.184 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.203+0000 7fcc17fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:34.184 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.204+0000 7fcc17fff640 1 -- 172.21.15.114:0/3680840789 <== mon.2 v1:172.21.15.114:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcc18004a90 con 0x7fcc2402a5a0 2024-03-29T11:23:34.184 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.204+0000 7fcc17fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:34.184 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.204+0000 7fcc17fff640 1 -- 172.21.15.114:0/3680840789 <== mon.2 v1:172.21.15.114:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fcc18004da0 con 0x7fcc2402a5a0 2024-03-29T11:23:34.184 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.204+0000 7fcc17fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:34.185 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.204+0000 7fcc17fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:23:34.185 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.204+0000 7fcc17fff640 10 monclient: dump: 2024-03-29T11:23:34.185 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:23:34.185 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:23:34.185 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:23:34.185 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:23:34.185 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:34.185 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:23:34.185 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:23:34.185 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:23:34.185 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:23:34.185 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:23:34.185 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.204+0000 7fcc297fa640 4 set_mon_vals no callback set 2024-03-29T11:23:34.185 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.204+0000 7fcc309ec640 5 monclient: authenticate success, global_id 4325 2024-03-29T11:23:34.185 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.204+0000 7fcc309ec640 10 monclient: _renew_subs 2024-03-29T11:23:34.185 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.204+0000 7fcc309ec640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:23:34.186 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.204+0000 7fcc309ec640 1 -- 172.21.15.114:0/3680840789 --> v1:172.21.15.114:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc240d4b30 con 0x7fcc2402a5a0 2024-03-29T11:23:34.186 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.204+0000 7fcc309ec640 10 client.4325.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:23:34.186 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.204+0000 7fcc309ec640 10 monclient: _renew_subs 2024-03-29T11:23:34.186 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.204+0000 7fcc309ec640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:23:34.186 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.204+0000 7fcc309ec640 1 -- 172.21.15.114:0/3680840789 --> v1:172.21.15.114:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc240d7390 con 0x7fcc2402a5a0 2024-03-29T11:23:34.186 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.204+0000 7fcc309ec640 1 librados: init done 2024-03-29T11:23:34.187 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.205+0000 7fcc309ec640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:34.187 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.205+0000 7fcc309ec640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:34.187 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.205+0000 7fcc309ec640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:23:34.188 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.205+0000 7fcc309ec640 1 -- 172.21.15.114:0/3680840789 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc24004cd0 con 0x7fcc2402a5a0 2024-03-29T11:23:34.188 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.207+0000 7fcc17fff640 1 -- 172.21.15.114:0/3680840789 <== mon.2 v1:172.21.15.114:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1594292461 0 0) 0x7fcc1801ad70 con 0x7fcc2402a5a0 2024-03-29T11:23:34.188 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.207+0000 7fcc17fff640 1 -- 172.21.15.114:0/3680840789 <== mon.2 v1:172.21.15.114:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3238987313 0 0) 0x7fcc18084a90 con 0x7fcc2402a5a0 2024-03-29T11:23:34.188 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.207+0000 7fcc17fff640 10 client.4325.objecter ms_dispatch 0x7fcc2c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:23:34.188 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.207+0000 7fcc17fff640 3 client.4325.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:23:34.188 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.207+0000 7fcc17fff640 3 client.4325.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:23:34.188 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.207+0000 7fcc17fff640 20 client.4325.objecter dump_active .. 0 homeless 2024-03-29T11:23:34.188 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.208+0000 7fcc2b7fe640 10 monclient: get_auth_request con 0x7fcc1006dcc0 auth_method 0 2024-03-29T11:23:34.188 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.208+0000 7fcc17fff640 10 client.4325.objecter ms_handle_connect 0x7fcc1006dcc0 2024-03-29T11:23:34.195 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.215+0000 7fcc17fff640 1 -- 172.21.15.114:0/3680840789 <== mon.2 v1:172.21.15.114:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fcc18050370 con 0x7fcc2402a5a0 2024-03-29T11:23:34.195 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.215+0000 7fcc17fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:34.195 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.215+0000 7fcc17fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:23:34.526 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.544+0000 7fcc309ec640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:34.526 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.545+0000 7fcc309ec640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:34.527 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.545+0000 7fcc309ec640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:23:34.527 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.545+0000 7fcc309ec640 1 -- 172.21.15.114:0/3680840789 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcc240d6ed0 con 0x7fcc2402a5a0 2024-03-29T11:23:34.527 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.548+0000 7fcc17fff640 1 -- 172.21.15.114:0/3680840789 <== mon.2 v1:172.21.15.114:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2858224613) 0x7fcc240d6ed0 con 0x7fcc2402a5a0 2024-03-29T11:23:34.527 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.548+0000 7fcc17fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:34.527 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.548+0000 7fcc17fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:23:34.531 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T11:23:34.531 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"b0de2b9b-c45a-44c7-b6ee-e83b39a1e108","created":"2024-03-29T11:20:49.420660+0000","modified":"2024-03-29T11:20:51.700167+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:51.700167+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a64785d3-6b11-499a-b706-cc827ab96a74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5882dadb-e7a8-4604-ba9a-418013c106c8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d7e8c409-bc66-4e4a-bbdb-420bad31388e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0682989-3595-4cd6-90a5-ab9111c278e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"67068120-e3ff-4817-bd06-44feb7479a35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09e8b-8a85-4997-bdea-75d754e6ee67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2f0d9ead-f47f-4cb3-92a4-0a5ad23cd831","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fdb6a25f-6bd8-4c65-9b9b-b460ba7673a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:23:34.534 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.555+0000 7fcc309ec640 10 librados: watch_flush enter 2024-03-29T11:23:34.534 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.555+0000 7fcc309ec640 10 librados: watch_flush exit 2024-03-29T11:23:34.534 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.555+0000 7fcc309ec640 20 client.4325.objecter shutdown clearing up homeless session... 2024-03-29T11:23:34.535 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.555+0000 7fcc309ec640 10 client.4325.objecter successfully canceled tick 2024-03-29T11:23:34.535 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.556+0000 7fcc309ec640 1 -- 172.21.15.114:0/3680840789 >> v1:172.21.15.114:6816/4038755055 conn(0x7fcc1006dcc0 legacy=0x7fcc10070180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:34.535 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.556+0000 7fcc309ec640 10 monclient: shutdown 2024-03-29T11:23:34.535 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.556+0000 7fcc309ec640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:34.535 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.556+0000 7fcc309ec640 1 -- 172.21.15.114:0/3680840789 >> v1:172.21.15.114:6790/0 conn(0x7fcc2402a5a0 legacy=0x7fcc24031290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:34.536 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.556+0000 7fcc309ec640 1 -- 172.21.15.114:0/3680840789 shutdown_connections 2024-03-29T11:23:34.536 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.556+0000 7fcc309ec640 1 -- 172.21.15.114:0/3680840789 >> 172.21.15.114:0/3680840789 conn(0x7fcc240058f0 msgr2=0x7fcc24034f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:34.536 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.557+0000 7fcc309ec640 1 -- 172.21.15.114:0/3680840789 shutdown_connections 2024-03-29T11:23:34.536 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.557+0000 7fcc309ec640 1 -- 172.21.15.114:0/3680840789 wait complete. 2024-03-29T11:23:34.536 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:34.557+0000 7fcc309ec640 1 librados: shutdown 2024-03-29T11:23:34.552 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:23:41.358 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:23:41.510 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.529+0000 7ffb2359e640 10 monclient: get_monmap_and_config 2024-03-29T11:23:41.510 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.530+0000 7ffb2359e640 10 monclient: build_initial_monmap 2024-03-29T11:23:41.510 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.530+0000 7ffb2359e640 10 monclient: monmap: 2024-03-29T11:23:41.510 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:23:41.510 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:23:41.510 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:23:41.531077+0000 2024-03-29T11:23:41.510 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:23:41.531077+0000 2024-03-29T11:23:41.510 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:41.510 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:23:41.510 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:23:41.510 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T11:23:41.510 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T11:23:41.511 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:23:41.516 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.533+0000 7ffb2359e640 1 Processor -- start 2024-03-29T11:23:41.516 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.534+0000 7ffb2359e640 1 -- start start 2024-03-29T11:23:41.516 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.534+0000 7ffb2359e640 10 monclient: init 2024-03-29T11:23:41.516 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.535+0000 7ffb2359e640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:41.516 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.535+0000 7ffb2359e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:23:41.517 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.535+0000 7ffb2359e640 10 monclient(hunting): picked mon.noname-c con 0x7ffb240af390 addr v1:172.21.15.176:6789/0 2024-03-29T11:23:41.517 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.535+0000 7ffb2359e640 10 monclient(hunting): picked mon.noname-a con 0x7ffb240b3990 addr v1:172.21.15.114:6789/0 2024-03-29T11:23:41.518 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.538+0000 7ffb2359e640 10 monclient(hunting): picked mon.noname-b con 0x7ffb240b16d0 addr v1:172.21.15.114:6790/0 2024-03-29T11:23:41.518 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.538+0000 7ffb2359e640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb240b47f0 con 0x7ffb240b3990 2024-03-29T11:23:41.519 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.538+0000 7ffb2359e640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb240b1b20 con 0x7ffb240af390 2024-03-29T11:23:41.519 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.538+0000 7ffb2359e640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb240b44a0 con 0x7ffb240b16d0 2024-03-29T11:23:41.519 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.538+0000 7ffb2359e640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:41.519 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.538+0000 7ffb2359e640 10 monclient(hunting): authenticate will time out at 1091.720711s 2024-03-29T11:23:41.522 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.541+0000 7ffb2259c640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7ffb240af390 0x7ffb241459f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.114:37329/0 (socket says 172.21.15.114:37329) 2024-03-29T11:23:41.522 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.541+0000 7ffb2259c640 1 -- 172.21.15.114:0/4218425177 learned_addr learned my addr 172.21.15.114:0/4218425177 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:23:41.523 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.543+0000 7ffb2159a640 1 -- 172.21.15.114:0/4218425177 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1725960214 0 0) 0x7ffb240b1b20 con 0x7ffb240af390 2024-03-29T11:23:41.524 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.543+0000 7ffb2159a640 10 monclient(hunting): _negotiate 2024-03-29T11:23:41.524 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.543+0000 7ffb2159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:41.524 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.543+0000 7ffb2159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:41.524 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.543+0000 7ffb2159a640 10 monclient(hunting): my global_id is 4330 2024-03-29T11:23:41.525 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.543+0000 7ffb2159a640 1 -- 172.21.15.114:0/4218425177 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb0c003520 con 0x7ffb240af390 2024-03-29T11:23:41.525 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.545+0000 7ffb2159a640 1 -- 172.21.15.114:0/4218425177 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2891407720 0 0) 0x7ffb240b47f0 con 0x7ffb240b3990 2024-03-29T11:23:41.525 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.545+0000 7ffb2159a640 10 monclient(hunting): _negotiate 2024-03-29T11:23:41.525 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.545+0000 7ffb2159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:41.525 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.545+0000 7ffb2159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:41.526 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.545+0000 7ffb2159a640 10 monclient(hunting): my global_id is 4329 2024-03-29T11:23:41.526 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.545+0000 7ffb2159a640 1 -- 172.21.15.114:0/4218425177 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb240b1b20 con 0x7ffb240b3990 2024-03-29T11:23:41.526 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.545+0000 7ffb2159a640 1 -- 172.21.15.114:0/4218425177 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 785246917 0 0) 0x7ffb0c003520 con 0x7ffb240af390 2024-03-29T11:23:41.526 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.545+0000 7ffb2159a640 1 -- 172.21.15.114:0/4218425177 >> v1:172.21.15.114:6790/0 conn(0x7ffb240b16d0 legacy=0x7ffb24151d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:41.526 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.545+0000 7ffb2159a640 1 -- 172.21.15.114:0/4218425177 >> v1:172.21.15.114:6789/0 conn(0x7ffb240b3990 legacy=0x7ffb240b0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:41.526 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.545+0000 7ffb2159a640 10 monclient: _finish_hunting 0 2024-03-29T11:23:41.526 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.545+0000 7ffb2159a640 1 monclient: found mon.noname-c 2024-03-29T11:23:41.526 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.545+0000 7ffb2159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:41.526 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.545+0000 7ffb2159a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.176:6789/0 2024-03-29T11:23:41.526 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.545+0000 7ffb2159a640 1 -- 172.21.15.114:0/4218425177 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb240b4c80 con 0x7ffb240af390 2024-03-29T11:23:41.526 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.545+0000 7ffb2159a640 10 monclient: _finish_auth 0 2024-03-29T11:23:41.526 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.545+0000 7ffb2159a640 10 monclient: _check_auth_tickets 2024-03-29T11:23:41.526 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.545+0000 7ffb2159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:41.526 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.545+0000 7ffb2159a640 1 -- 172.21.15.114:0/4218425177 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7ffb14001610 con 0x7ffb240af390 2024-03-29T11:23:41.527 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.545+0000 7ffb2159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:41.527 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.545+0000 7ffb2159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:23:41.527 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.545+0000 7ffb2159a640 10 monclient: dump: 2024-03-29T11:23:41.527 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:23:41.527 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:23:41.527 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:23:41.527 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:23:41.527 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:41.527 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:23:41.527 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:23:41.527 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:23:41.527 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:23:41.527 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:23:41.527 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.545+0000 7ffb2359e640 5 monclient: authenticate success, global_id 4330 2024-03-29T11:23:41.527 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.545+0000 7ffb2359e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:41.529 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.547+0000 7ffb2159a640 1 -- 172.21.15.114:0/4218425177 <== mon.1 v1:172.21.15.176:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffb14001770 con 0x7ffb240af390 2024-03-29T11:23:41.529 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.547+0000 7ffb2159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:41.529 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.547+0000 7ffb2159a640 1 -- 172.21.15.114:0/4218425177 <== mon.1 v1:172.21.15.176:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7ffb14001b80 con 0x7ffb240af390 2024-03-29T11:23:41.529 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.547+0000 7ffb2159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:41.529 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.547+0000 7ffb2159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:23:41.529 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.547+0000 7ffb2159a640 10 monclient: dump: 2024-03-29T11:23:41.529 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:23:41.529 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:23:41.529 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:23:41.530 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:23:41.530 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:41.530 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:23:41.530 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:23:41.530 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:23:41.530 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:23:41.530 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:23:41.530 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.547+0000 7ffb2359e640 10 monclient: get_monmap_and_config success 2024-03-29T11:23:41.530 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.547+0000 7ffb2359e640 4 set_mon_vals no callback set 2024-03-29T11:23:41.530 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.548+0000 7ffb2359e640 10 monclient: shutdown 2024-03-29T11:23:41.530 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.548+0000 7ffb2359e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:41.530 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.548+0000 7ffb2359e640 1 -- 172.21.15.114:0/4218425177 >> v1:172.21.15.176:6789/0 conn(0x7ffb240af390 legacy=0x7ffb241459f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:41.530 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.549+0000 7ffb2359e640 1 -- 172.21.15.114:0/4218425177 shutdown_connections 2024-03-29T11:23:41.530 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.549+0000 7ffb2359e640 1 -- 172.21.15.114:0/4218425177 >> 172.21.15.114:0/4218425177 conn(0x7ffb2406d230 msgr2=0x7ffb24066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:41.531 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.552+0000 7ffb2359e640 1 -- 172.21.15.114:0/4218425177 shutdown_connections 2024-03-29T11:23:41.531 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.552+0000 7ffb2359e640 1 -- 172.21.15.114:0/4218425177 wait complete. 2024-03-29T11:23:41.534 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.553+0000 7ffb2359e640 10 monclient: build_initial_monmap 2024-03-29T11:23:41.535 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.553+0000 7ffb2359e640 10 monclient: monmap: 2024-03-29T11:23:41.535 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:23:41.535 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:23:41.535 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T11:23:41.535 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T11:23:41.535 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:41.535 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:23:41.535 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:23:41.535 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T11:23:41.535 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T11:23:41.535 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:23:41.535 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.554+0000 7ffb2359e640 1 librados: starting msgr at 2024-03-29T11:23:41.535 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.554+0000 7ffb2359e640 1 librados: starting objecter 2024-03-29T11:23:41.535 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.554+0000 7ffb2359e640 1 Processor -- start 2024-03-29T11:23:41.536 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.554+0000 7ffb2359e640 1 -- start start 2024-03-29T11:23:41.536 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.554+0000 7ffb2359e640 1 librados: setting wanted keys 2024-03-29T11:23:41.536 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.554+0000 7ffb2359e640 1 librados: calling monclient init 2024-03-29T11:23:41.536 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.554+0000 7ffb2359e640 10 monclient: init 2024-03-29T11:23:41.536 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.555+0000 7ffb2359e640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:41.536 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.555+0000 7ffb2359e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:23:41.536 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.555+0000 7ffb2359e640 10 monclient(hunting): picked mon.noname-a con 0x7ffb240af390 addr v1:172.21.15.114:6789/0 2024-03-29T11:23:41.536 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.555+0000 7ffb2359e640 10 monclient(hunting): picked mon.noname-c con 0x7ffb240b16d0 addr v1:172.21.15.114:6790/0 2024-03-29T11:23:41.538 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.555+0000 7ffb2359e640 10 monclient(hunting): picked mon.noname-b con 0x7ffb240b3990 addr v1:172.21.15.176:6789/0 2024-03-29T11:23:41.538 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.558+0000 7ffb2359e640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb2406f590 con 0x7ffb240af390 2024-03-29T11:23:41.538 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.558+0000 7ffb2359e640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb2406f800 con 0x7ffb240b3990 2024-03-29T11:23:41.538 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.558+0000 7ffb2359e640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb2406fae0 con 0x7ffb240b16d0 2024-03-29T11:23:41.538 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.558+0000 7ffb2359e640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:41.538 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.558+0000 7ffb2359e640 10 monclient(hunting): authenticate will time out at 1091.740498s 2024-03-29T11:23:41.538 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.558+0000 7ffb22d9d640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7ffb240b3990 0x7ffb241e5f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.114:37451/0 (socket says 172.21.15.114:37451) 2024-03-29T11:23:41.538 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.558+0000 7ffb22d9d640 1 -- 172.21.15.114:0/2697143199 learned_addr learned my addr 172.21.15.114:0/2697143199 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:23:41.538 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.558+0000 7ffb12ffd640 10 client.?.objecter ms_handle_connect 0x7ffb240b3990 2024-03-29T11:23:41.539 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.558+0000 7ffb12ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:41.542 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.561+0000 7ffb12ffd640 1 -- 172.21.15.114:0/2697143199 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2262847432 0 0) 0x7ffb2406f800 con 0x7ffb240b3990 2024-03-29T11:23:41.542 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.561+0000 7ffb12ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:23:41.542 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.561+0000 7ffb12ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:41.542 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.561+0000 7ffb12ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:41.542 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.561+0000 7ffb12ffd640 10 monclient(hunting): my global_id is 4333 2024-03-29T11:23:41.542 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.561+0000 7ffb12ffd640 1 -- 172.21.15.114:0/2697143199 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb04004eb0 con 0x7ffb240b3990 2024-03-29T11:23:41.542 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.561+0000 7ffb12ffd640 1 -- 172.21.15.114:0/2697143199 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 18081553 0 0) 0x7ffb04004eb0 con 0x7ffb240b3990 2024-03-29T11:23:41.542 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.561+0000 7ffb12ffd640 1 -- 172.21.15.114:0/2697143199 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffb2406f800 con 0x7ffb240b3990 2024-03-29T11:23:41.542 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.561+0000 7ffb12ffd640 1 -- 172.21.15.114:0/2697143199 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7ffb1c001c60 con 0x7ffb240b3990 2024-03-29T11:23:41.542 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.561+0000 7ffb12ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:41.542 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.561+0000 7ffb12ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:23:41.542 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.561+0000 7ffb12ffd640 10 monclient(hunting): dump: 2024-03-29T11:23:41.542 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:23:41.543 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:23:41.543 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:23:41.543 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:23:41.543 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:41.543 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:23:41.543 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:23:41.543 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:23:41.543 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:23:41.543 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:23:41.543 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.561+0000 7ffb12ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:23:41.543 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.562+0000 7ffb12ffd640 10 client.?.objecter ms_handle_connect 0x7ffb240af390 2024-03-29T11:23:41.543 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.562+0000 7ffb12ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:41.543 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.562+0000 7ffb12ffd640 1 -- 172.21.15.114:0/2697143199 <== mon.1 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1853567298 0 0) 0x7ffb2406f800 con 0x7ffb240b3990 2024-03-29T11:23:41.543 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.562+0000 7ffb12ffd640 1 -- 172.21.15.114:0/2697143199 >> v1:172.21.15.114:6790/0 conn(0x7ffb240b16d0 legacy=0x7ffb24070fb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:23:41.545 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.564+0000 7ffb12ffd640 1 -- 172.21.15.114:0/2697143199 >> v1:172.21.15.114:6789/0 conn(0x7ffb240af390 legacy=0x7ffb240708a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:41.545 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.564+0000 7ffb12ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:23:41.546 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.564+0000 7ffb12ffd640 1 monclient: found mon.b 2024-03-29T11:23:41.546 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.564+0000 7ffb12ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:41.546 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.564+0000 7ffb12ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:23:41.546 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.564+0000 7ffb12ffd640 1 -- 172.21.15.114:0/2697143199 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb2406fdc0 con 0x7ffb240b3990 2024-03-29T11:23:41.546 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.565+0000 7ffb12ffd640 10 monclient: _finish_auth 0 2024-03-29T11:23:41.546 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.565+0000 7ffb12ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:23:41.546 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.565+0000 7ffb12ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:41.546 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.565+0000 7ffb2359e640 5 monclient: authenticate success, global_id 4333 2024-03-29T11:23:41.546 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.565+0000 7ffb2359e640 10 monclient: _renew_subs 2024-03-29T11:23:41.546 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.565+0000 7ffb2359e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:23:41.546 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.565+0000 7ffb2359e640 1 -- 172.21.15.114:0/2697143199 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb24070380 con 0x7ffb240b3990 2024-03-29T11:23:41.546 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.565+0000 7ffb2359e640 10 client.4333.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:23:41.546 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.565+0000 7ffb2359e640 10 monclient: _renew_subs 2024-03-29T11:23:41.546 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.565+0000 7ffb2359e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:23:41.546 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.565+0000 7ffb2359e640 1 -- 172.21.15.114:0/2697143199 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb2414a510 con 0x7ffb240b3990 2024-03-29T11:23:41.547 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.565+0000 7ffb2359e640 1 librados: init done 2024-03-29T11:23:41.547 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.566+0000 7ffb12ffd640 1 -- 172.21.15.114:0/2697143199 <== mon.1 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffb1c003990 con 0x7ffb240b3990 2024-03-29T11:23:41.547 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.566+0000 7ffb12ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:41.547 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.566+0000 7ffb20d99640 4 set_mon_vals no callback set 2024-03-29T11:23:41.547 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.567+0000 7ffb2359e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:41.548 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.567+0000 7ffb12ffd640 1 -- 172.21.15.114:0/2697143199 <== mon.1 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7ffb1c003da0 con 0x7ffb240b3990 2024-03-29T11:23:41.548 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.567+0000 7ffb12ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:41.548 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.567+0000 7ffb12ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:23:41.548 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.567+0000 7ffb12ffd640 10 monclient: dump: 2024-03-29T11:23:41.548 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:23:41.548 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:23:41.548 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:23:41.548 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:23:41.548 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:41.548 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:23:41.548 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:23:41.548 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:23:41.548 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:23:41.548 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:23:41.550 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.568+0000 7ffb12ffd640 1 -- 172.21.15.114:0/2697143199 <== mon.1 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1594292461 0 0) 0x7ffb1c003f00 con 0x7ffb240b3990 2024-03-29T11:23:41.550 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.569+0000 7ffb2359e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:41.550 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.569+0000 7ffb2359e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:23:41.550 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.569+0000 7ffb2359e640 1 -- 172.21.15.114:0/2697143199 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb08005080 con 0x7ffb240b3990 2024-03-29T11:23:41.550 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.569+0000 7ffb2259c640 10 monclient: get_auth_request con 0x7ffb0406db50 auth_method 0 2024-03-29T11:23:41.550 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.569+0000 7ffb12ffd640 1 -- 172.21.15.114:0/2697143199 <== mon.1 v1:172.21.15.176:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3238987313 0 0) 0x7ffb1c084e90 con 0x7ffb240b3990 2024-03-29T11:23:41.550 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.569+0000 7ffb12ffd640 10 client.4333.objecter ms_dispatch 0x7ffb24000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:23:41.550 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.569+0000 7ffb12ffd640 3 client.4333.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:23:41.550 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.569+0000 7ffb12ffd640 3 client.4333.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:23:41.550 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.569+0000 7ffb12ffd640 20 client.4333.objecter dump_active .. 0 homeless 2024-03-29T11:23:41.550 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.569+0000 7ffb12ffd640 10 client.4333.objecter ms_handle_connect 0x7ffb0406db50 2024-03-29T11:23:41.559 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.579+0000 7ffb12ffd640 1 -- 172.21.15.114:0/2697143199 <== mon.1 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ffb1c050770 con 0x7ffb240b3990 2024-03-29T11:23:41.559 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.579+0000 7ffb12ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:41.559 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.579+0000 7ffb12ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:23:41.881 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.901+0000 7ffb2359e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:41.881 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.901+0000 7ffb2359e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:41.881 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.901+0000 7ffb2359e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:23:41.881 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.901+0000 7ffb2359e640 1 -- 172.21.15.114:0/2697143199 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffb08005330 con 0x7ffb240b3990 2024-03-29T11:23:41.882 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.902+0000 7ffb12ffd640 1 -- 172.21.15.114:0/2697143199 <== mon.1 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2858224613) 0x7ffb1c0526a0 con 0x7ffb240b3990 2024-03-29T11:23:41.882 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.902+0000 7ffb12ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:41.882 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.902+0000 7ffb12ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:23:41.883 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T11:23:41.883 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"b0de2b9b-c45a-44c7-b6ee-e83b39a1e108","created":"2024-03-29T11:20:49.420660+0000","modified":"2024-03-29T11:20:51.700167+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:51.700167+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a64785d3-6b11-499a-b706-cc827ab96a74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5882dadb-e7a8-4604-ba9a-418013c106c8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d7e8c409-bc66-4e4a-bbdb-420bad31388e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0682989-3595-4cd6-90a5-ab9111c278e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"67068120-e3ff-4817-bd06-44feb7479a35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09e8b-8a85-4997-bdea-75d754e6ee67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2f0d9ead-f47f-4cb3-92a4-0a5ad23cd831","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fdb6a25f-6bd8-4c65-9b9b-b460ba7673a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:23:41.886 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.906+0000 7ffb2359e640 10 librados: watch_flush enter 2024-03-29T11:23:41.886 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.906+0000 7ffb2359e640 10 librados: watch_flush exit 2024-03-29T11:23:41.887 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.906+0000 7ffb2359e640 20 client.4333.objecter shutdown clearing up homeless session... 2024-03-29T11:23:41.887 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.906+0000 7ffb2359e640 10 client.4333.objecter successfully canceled tick 2024-03-29T11:23:41.887 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.907+0000 7ffb2359e640 1 -- 172.21.15.114:0/2697143199 >> v1:172.21.15.114:6816/4038755055 conn(0x7ffb0406db50 legacy=0x7ffb04070010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:41.887 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.907+0000 7ffb2359e640 10 monclient: shutdown 2024-03-29T11:23:41.887 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.907+0000 7ffb2359e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:41.887 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.907+0000 7ffb2359e640 1 -- 172.21.15.114:0/2697143199 >> v1:172.21.15.176:6789/0 conn(0x7ffb240b3990 legacy=0x7ffb241e5f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:41.891 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.910+0000 7ffb2359e640 1 -- 172.21.15.114:0/2697143199 shutdown_connections 2024-03-29T11:23:41.891 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.910+0000 7ffb2359e640 1 -- 172.21.15.114:0/2697143199 >> 172.21.15.114:0/2697143199 conn(0x7ffb2406d230 msgr2=0x7ffb240b0fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:41.891 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.911+0000 7ffb2359e640 1 -- 172.21.15.114:0/2697143199 shutdown_connections 2024-03-29T11:23:41.891 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.911+0000 7ffb2359e640 1 -- 172.21.15.114:0/2697143199 wait complete. 2024-03-29T11:23:41.891 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:41.912+0000 7ffb2359e640 1 librados: shutdown 2024-03-29T11:23:41.908 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:23:48.715 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:23:48.866 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.886+0000 7fbd15043640 10 monclient: get_monmap_and_config 2024-03-29T11:23:48.867 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.886+0000 7fbd15043640 10 monclient: build_initial_monmap 2024-03-29T11:23:48.867 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.886+0000 7fbd15043640 10 monclient: monmap: 2024-03-29T11:23:48.867 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:23:48.867 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:23:48.867 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:23:48.887544+0000 2024-03-29T11:23:48.867 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:23:48.887544+0000 2024-03-29T11:23:48.867 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:48.867 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:23:48.867 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:23:48.867 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T11:23:48.867 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T11:23:48.867 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:23:48.870 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.889+0000 7fbd15043640 1 Processor -- start 2024-03-29T11:23:48.870 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.889+0000 7fbd15043640 1 -- start start 2024-03-29T11:23:48.870 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.889+0000 7fbd15043640 10 monclient: init 2024-03-29T11:23:48.870 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.890+0000 7fbd15043640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:48.871 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.890+0000 7fbd15043640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:23:48.871 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.890+0000 7fbd15043640 10 monclient(hunting): picked mon.noname-a con 0x7fbd100b4330 addr v1:172.21.15.114:6789/0 2024-03-29T11:23:48.871 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.890+0000 7fbd15043640 10 monclient(hunting): picked mon.noname-b con 0x7fbd101470b0 addr v1:172.21.15.114:6790/0 2024-03-29T11:23:48.876 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.895+0000 7fbd15043640 10 monclient(hunting): picked mon.noname-c con 0x7fbd101439b0 addr v1:172.21.15.176:6789/0 2024-03-29T11:23:48.876 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.895+0000 7fbd15043640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd1006f2a0 con 0x7fbd100b4330 2024-03-29T11:23:48.876 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.895+0000 7fbd15043640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd10146b90 con 0x7fbd101439b0 2024-03-29T11:23:48.876 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.895+0000 7fbd15043640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd100b2ca0 con 0x7fbd101470b0 2024-03-29T11:23:48.876 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.895+0000 7fbd15043640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:48.876 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.895+0000 7fbd15043640 10 monclient(hunting): authenticate will time out at 1099.077346s 2024-03-29T11:23:48.876 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.896+0000 7fbd0ed9d640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fbd100b4330 0x7fbd100b4710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:41575/0 (socket says 172.21.15.114:41575) 2024-03-29T11:23:48.876 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.896+0000 7fbd0ed9d640 1 -- 172.21.15.114:0/421405995 learned_addr learned my addr 172.21.15.114:0/421405995 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:23:48.876 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.896+0000 7fbd0dd9b640 1 -- 172.21.15.114:0/421405995 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 466196586 0 0) 0x7fbd100b2ca0 con 0x7fbd101470b0 2024-03-29T11:23:48.880 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.896+0000 7fbd0dd9b640 10 monclient(hunting): _negotiate 2024-03-29T11:23:48.880 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.896+0000 7fbd0dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:48.880 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.896+0000 7fbd0dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:48.880 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.896+0000 7fbd0dd9b640 10 monclient(hunting): my global_id is 4331 2024-03-29T11:23:48.880 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.896+0000 7fbd0dd9b640 1 -- 172.21.15.114:0/421405995 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbcf8003520 con 0x7fbd101470b0 2024-03-29T11:23:48.880 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.897+0000 7fbd0dd9b640 1 -- 172.21.15.114:0/421405995 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1299692713 0 0) 0x7fbcf8003520 con 0x7fbd101470b0 2024-03-29T11:23:48.881 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.897+0000 7fbd0dd9b640 1 -- 172.21.15.114:0/421405995 >> v1:172.21.15.176:6789/0 conn(0x7fbd101439b0 legacy=0x7fbd10151d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:48.881 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.897+0000 7fbd0dd9b640 1 -- 172.21.15.114:0/421405995 >> v1:172.21.15.114:6789/0 conn(0x7fbd100b4330 legacy=0x7fbd100b4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:48.881 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.897+0000 7fbd0dd9b640 10 monclient: _finish_hunting 0 2024-03-29T11:23:48.881 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.897+0000 7fbd0dd9b640 1 monclient: found mon.noname-b 2024-03-29T11:23:48.881 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.897+0000 7fbd0dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:48.881 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.897+0000 7fbd0dd9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.114:6790/0 2024-03-29T11:23:48.881 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.897+0000 7fbd0dd9b640 1 -- 172.21.15.114:0/421405995 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd1006e0c0 con 0x7fbd101470b0 2024-03-29T11:23:48.881 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.897+0000 7fbd0dd9b640 10 monclient: _finish_auth 0 2024-03-29T11:23:48.881 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.897+0000 7fbd0dd9b640 10 monclient: _check_auth_tickets 2024-03-29T11:23:48.881 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.897+0000 7fbd0dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:48.882 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.897+0000 7fbd0dd9b640 1 -- 172.21.15.114:0/421405995 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fbd00001690 con 0x7fbd101470b0 2024-03-29T11:23:48.882 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.897+0000 7fbd0dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:48.882 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.897+0000 7fbd0dd9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:23:48.882 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.897+0000 7fbd0dd9b640 10 monclient: dump: 2024-03-29T11:23:48.882 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:23:48.882 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:23:48.882 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:23:48.882 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:23:48.883 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:48.883 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:23:48.883 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:23:48.883 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:23:48.883 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:23:48.883 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:23:48.883 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.897+0000 7fbd15043640 5 monclient: authenticate success, global_id 4331 2024-03-29T11:23:48.883 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.897+0000 7fbd15043640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:48.883 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.897+0000 7fbd0dd9b640 1 -- 172.21.15.114:0/421405995 <== mon.2 v1:172.21.15.114:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbd000017f0 con 0x7fbd101470b0 2024-03-29T11:23:48.883 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.897+0000 7fbd0dd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:48.884 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.897+0000 7fbd0dd9b640 1 -- 172.21.15.114:0/421405995 <== mon.2 v1:172.21.15.114:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fbd00001c40 con 0x7fbd101470b0 2024-03-29T11:23:48.884 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.897+0000 7fbd0dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:48.884 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.897+0000 7fbd0dd9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:23:48.884 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.897+0000 7fbd0dd9b640 10 monclient: dump: 2024-03-29T11:23:48.884 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:23:48.884 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:23:48.884 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:23:48.884 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:23:48.884 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:48.884 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:23:48.885 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:23:48.885 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:23:48.885 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:23:48.885 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:23:48.885 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.898+0000 7fbd15043640 10 monclient: get_monmap_and_config success 2024-03-29T11:23:48.885 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.898+0000 7fbd15043640 4 set_mon_vals no callback set 2024-03-29T11:23:48.886 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.899+0000 7fbd15043640 10 monclient: shutdown 2024-03-29T11:23:48.886 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.899+0000 7fbd15043640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:48.886 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.899+0000 7fbd15043640 1 -- 172.21.15.114:0/421405995 >> v1:172.21.15.114:6790/0 conn(0x7fbd101470b0 legacy=0x7fbd100b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:48.887 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.901+0000 7fbd15043640 1 -- 172.21.15.114:0/421405995 shutdown_connections 2024-03-29T11:23:48.887 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.901+0000 7fbd15043640 1 -- 172.21.15.114:0/421405995 >> 172.21.15.114:0/421405995 conn(0x7fbd10066a00 msgr2=0x7fbd10066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:48.887 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.903+0000 7fbd15043640 1 -- 172.21.15.114:0/421405995 shutdown_connections 2024-03-29T11:23:48.887 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.905+0000 7fbd15043640 1 -- 172.21.15.114:0/421405995 wait complete. 2024-03-29T11:23:48.888 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.907+0000 7fbd15043640 10 monclient: build_initial_monmap 2024-03-29T11:23:48.888 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.907+0000 7fbd15043640 10 monclient: monmap: 2024-03-29T11:23:48.888 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:23:48.888 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:23:48.888 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T11:23:48.888 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T11:23:48.889 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:48.889 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:23:48.889 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:23:48.889 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T11:23:48.889 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T11:23:48.889 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:23:48.889 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.907+0000 7fbd15043640 1 librados: starting msgr at 2024-03-29T11:23:48.889 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.907+0000 7fbd15043640 1 librados: starting objecter 2024-03-29T11:23:48.889 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.907+0000 7fbd15043640 1 Processor -- start 2024-03-29T11:23:48.889 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.907+0000 7fbd15043640 1 -- start start 2024-03-29T11:23:48.889 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.907+0000 7fbd15043640 1 librados: setting wanted keys 2024-03-29T11:23:48.890 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.907+0000 7fbd15043640 1 librados: calling monclient init 2024-03-29T11:23:48.890 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.907+0000 7fbd15043640 10 monclient: init 2024-03-29T11:23:48.890 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.908+0000 7fbd15043640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:48.890 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.908+0000 7fbd15043640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:23:48.890 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.908+0000 7fbd15043640 10 monclient(hunting): picked mon.noname-b con 0x7fbd100b4330 addr v1:172.21.15.176:6789/0 2024-03-29T11:23:48.890 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.908+0000 7fbd15043640 10 monclient(hunting): picked mon.noname-a con 0x7fbd101439b0 addr v1:172.21.15.114:6789/0 2024-03-29T11:23:48.890 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.908+0000 7fbd15043640 10 monclient(hunting): picked mon.noname-c con 0x7fbd101470b0 addr v1:172.21.15.114:6790/0 2024-03-29T11:23:48.890 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.908+0000 7fbd15043640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd101419a0 con 0x7fbd101439b0 2024-03-29T11:23:48.890 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.908+0000 7fbd15043640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd10121270 con 0x7fbd100b4330 2024-03-29T11:23:48.890 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.908+0000 7fbd15043640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd10071740 con 0x7fbd101470b0 2024-03-29T11:23:48.891 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.908+0000 7fbd15043640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:48.891 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.908+0000 7fbd15043640 10 monclient(hunting): authenticate will time out at 1099.090865s 2024-03-29T11:23:48.894 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.912+0000 7fbd0ed9d640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7fbd100b4330 0x7fbd1011f850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.114:53145/0 (socket says 172.21.15.114:53145) 2024-03-29T11:23:48.894 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.912+0000 7fbd0ed9d640 1 -- 172.21.15.114:0/200277826 learned_addr learned my addr 172.21.15.114:0/200277826 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:23:48.894 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.912+0000 7fbcf77fe640 10 client.?.objecter ms_handle_connect 0x7fbd100b4330 2024-03-29T11:23:48.894 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.912+0000 7fbcf77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:48.895 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.912+0000 7fbcf77fe640 1 -- 172.21.15.114:0/200277826 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 789267717 0 0) 0x7fbd10121270 con 0x7fbd100b4330 2024-03-29T11:23:48.895 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.912+0000 7fbcf77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:23:48.895 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.912+0000 7fbcf77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:48.895 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.912+0000 7fbcf77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:48.895 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.912+0000 7fbcf77fe640 10 monclient(hunting): my global_id is 4336 2024-03-29T11:23:48.895 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.912+0000 7fbcf77fe640 1 -- 172.21.15.114:0/200277826 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbcfc004eb0 con 0x7fbd100b4330 2024-03-29T11:23:48.895 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.913+0000 7fbcf77fe640 1 -- 172.21.15.114:0/200277826 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2447662598 0 0) 0x7fbcfc004eb0 con 0x7fbd100b4330 2024-03-29T11:23:48.896 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.913+0000 7fbcf77fe640 1 -- 172.21.15.114:0/200277826 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbd10121270 con 0x7fbd100b4330 2024-03-29T11:23:48.896 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.913+0000 7fbcf77fe640 1 -- 172.21.15.114:0/200277826 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fbd04001890 con 0x7fbd100b4330 2024-03-29T11:23:48.896 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.913+0000 7fbcf77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:48.896 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.913+0000 7fbcf77fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:23:48.896 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.913+0000 7fbcf77fe640 10 monclient(hunting): dump: 2024-03-29T11:23:48.896 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:23:48.896 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:23:48.897 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:23:48.897 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:23:48.897 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:48.897 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:23:48.897 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:23:48.897 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:23:48.897 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:23:48.898 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:23:48.898 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.913+0000 7fbcf77fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:23:48.898 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.914+0000 7fbcf77fe640 10 client.?.objecter ms_handle_connect 0x7fbd101470b0 2024-03-29T11:23:48.898 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.914+0000 7fbcf77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:48.898 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.914+0000 7fbcf77fe640 1 -- 172.21.15.114:0/200277826 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2265379521 0 0) 0x7fbd10071740 con 0x7fbd101470b0 2024-03-29T11:23:48.898 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.914+0000 7fbcf77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:23:48.898 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.914+0000 7fbcf77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:48.899 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.914+0000 7fbcf77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:48.899 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.914+0000 7fbcf77fe640 10 monclient(hunting): my global_id is 4334 2024-03-29T11:23:48.899 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.914+0000 7fbcf77fe640 1 -- 172.21.15.114:0/200277826 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbcfc004eb0 con 0x7fbd101470b0 2024-03-29T11:23:48.901 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.914+0000 7fbcf77fe640 1 -- 172.21.15.114:0/200277826 <== mon.1 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 431477044 0 0) 0x7fbd10121270 con 0x7fbd100b4330 2024-03-29T11:23:48.901 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.914+0000 7fbcf77fe640 1 -- 172.21.15.114:0/200277826 >> v1:172.21.15.114:6790/0 conn(0x7fbd101470b0 legacy=0x7fbd10141290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:48.901 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.914+0000 7fbcf77fe640 1 -- 172.21.15.114:0/200277826 >> v1:172.21.15.114:6789/0 conn(0x7fbd101439b0 legacy=0x7fbd10140b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:48.901 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.914+0000 7fbcf77fe640 10 monclient: _finish_hunting 0 2024-03-29T11:23:48.901 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.914+0000 7fbcf77fe640 1 monclient: found mon.b 2024-03-29T11:23:48.901 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.914+0000 7fbcf77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:48.901 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.914+0000 7fbcf77fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:23:48.901 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.914+0000 7fbcf77fe640 1 -- 172.21.15.114:0/200277826 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd10071260 con 0x7fbd100b4330 2024-03-29T11:23:48.901 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.914+0000 7fbcf77fe640 10 monclient: _finish_auth 0 2024-03-29T11:23:48.901 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.914+0000 7fbcf77fe640 10 monclient: _check_auth_tickets 2024-03-29T11:23:48.901 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.914+0000 7fbcf77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:48.902 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.914+0000 7fbd15043640 5 monclient: authenticate success, global_id 4336 2024-03-29T11:23:48.902 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.914+0000 7fbd15043640 10 monclient: _renew_subs 2024-03-29T11:23:48.902 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.914+0000 7fbd15043640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:23:48.902 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.914+0000 7fbd15043640 1 -- 172.21.15.114:0/200277826 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd10071410 con 0x7fbd100b4330 2024-03-29T11:23:48.902 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.914+0000 7fbd15043640 10 client.4336.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:23:48.902 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.914+0000 7fbd15043640 10 monclient: _renew_subs 2024-03-29T11:23:48.902 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.914+0000 7fbd15043640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:23:48.902 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.914+0000 7fbd15043640 1 -- 172.21.15.114:0/200277826 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd10141f10 con 0x7fbd100b4330 2024-03-29T11:23:48.902 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.914+0000 7fbd15043640 1 librados: init done 2024-03-29T11:23:48.902 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.917+0000 7fbcf77fe640 1 -- 172.21.15.114:0/200277826 <== mon.1 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbd040019f0 con 0x7fbd100b4330 2024-03-29T11:23:48.902 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.917+0000 7fbcf77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:48.902 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.917+0000 7fbcf77fe640 1 -- 172.21.15.114:0/200277826 <== mon.1 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fbd04004a80 con 0x7fbd100b4330 2024-03-29T11:23:48.902 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.917+0000 7fbcf77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:48.902 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.917+0000 7fbcf77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:23:48.903 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.917+0000 7fbcf77fe640 10 monclient: dump: 2024-03-29T11:23:48.903 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:23:48.903 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:23:48.903 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:23:48.903 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:23:48.903 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:48.903 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:23:48.903 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:23:48.903 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:23:48.903 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:23:48.903 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:23:48.903 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.917+0000 7fbd0d59a640 4 set_mon_vals no callback set 2024-03-29T11:23:48.903 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.918+0000 7fbd15043640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:48.903 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.918+0000 7fbd15043640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:48.903 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.918+0000 7fbd15043640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:23:48.904 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.918+0000 7fbd15043640 1 -- 172.21.15.114:0/200277826 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcf0005080 con 0x7fbd100b4330 2024-03-29T11:23:48.904 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.924+0000 7fbcf77fe640 1 -- 172.21.15.114:0/200277826 <== mon.1 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1594292461 0 0) 0x7fbd04004be0 con 0x7fbd100b4330 2024-03-29T11:23:48.905 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.925+0000 7fbcf77fe640 1 -- 172.21.15.114:0/200277826 <== mon.1 v1:172.21.15.176:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3238987313 0 0) 0x7fbd04084b10 con 0x7fbd100b4330 2024-03-29T11:23:48.905 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.925+0000 7fbcf77fe640 10 client.4336.objecter ms_dispatch 0x7fbd10000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:23:48.905 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.925+0000 7fbcf77fe640 3 client.4336.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:23:48.905 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.925+0000 7fbcf77fe640 3 client.4336.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:23:48.905 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.925+0000 7fbcf77fe640 20 client.4336.objecter dump_active .. 0 homeless 2024-03-29T11:23:48.906 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.925+0000 7fbd0e59c640 10 monclient: get_auth_request con 0x7fbcfc06dd30 auth_method 0 2024-03-29T11:23:48.906 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.926+0000 7fbcf77fe640 10 client.4336.objecter ms_handle_connect 0x7fbcfc06dd30 2024-03-29T11:23:48.913 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.933+0000 7fbcf77fe640 1 -- 172.21.15.114:0/200277826 <== mon.1 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbd04050220 con 0x7fbd100b4330 2024-03-29T11:23:48.913 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.933+0000 7fbcf77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:48.913 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:48.933+0000 7fbcf77fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:23:49.237 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:49.257+0000 7fbd15043640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:49.237 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:49.257+0000 7fbd15043640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:49.237 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:49.257+0000 7fbd15043640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:23:49.237 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:49.257+0000 7fbd15043640 1 -- 172.21.15.114:0/200277826 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbcf0005330 con 0x7fbd100b4330 2024-03-29T11:23:49.240 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:49.260+0000 7fbcf77fe640 1 -- 172.21.15.114:0/200277826 <== mon.1 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2858224613) 0x7fbd04052320 con 0x7fbd100b4330 2024-03-29T11:23:49.240 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:49.260+0000 7fbcf77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:49.240 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:49.260+0000 7fbcf77fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:23:49.242 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T11:23:49.242 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"b0de2b9b-c45a-44c7-b6ee-e83b39a1e108","created":"2024-03-29T11:20:49.420660+0000","modified":"2024-03-29T11:20:51.700167+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:51.700167+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a64785d3-6b11-499a-b706-cc827ab96a74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5882dadb-e7a8-4604-ba9a-418013c106c8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d7e8c409-bc66-4e4a-bbdb-420bad31388e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0682989-3595-4cd6-90a5-ab9111c278e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"67068120-e3ff-4817-bd06-44feb7479a35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09e8b-8a85-4997-bdea-75d754e6ee67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2f0d9ead-f47f-4cb3-92a4-0a5ad23cd831","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fdb6a25f-6bd8-4c65-9b9b-b460ba7673a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:23:49.243 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:49.263+0000 7fbd15043640 10 librados: watch_flush enter 2024-03-29T11:23:49.244 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:49.263+0000 7fbd15043640 10 librados: watch_flush exit 2024-03-29T11:23:49.244 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:49.263+0000 7fbd15043640 20 client.4336.objecter shutdown clearing up homeless session... 2024-03-29T11:23:49.244 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:49.263+0000 7fbd15043640 10 client.4336.objecter successfully canceled tick 2024-03-29T11:23:49.244 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:49.264+0000 7fbd15043640 1 -- 172.21.15.114:0/200277826 >> v1:172.21.15.114:6816/4038755055 conn(0x7fbcfc06dd30 legacy=0x7fbcfc0701f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:49.244 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:49.264+0000 7fbd15043640 10 monclient: shutdown 2024-03-29T11:23:49.244 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:49.264+0000 7fbd15043640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:49.244 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:49.264+0000 7fbd15043640 1 -- 172.21.15.114:0/200277826 >> v1:172.21.15.176:6789/0 conn(0x7fbd100b4330 legacy=0x7fbd1011f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:49.244 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:49.264+0000 7fbd15043640 1 -- 172.21.15.114:0/200277826 shutdown_connections 2024-03-29T11:23:49.244 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:49.264+0000 7fbd15043640 1 -- 172.21.15.114:0/200277826 >> 172.21.15.114:0/200277826 conn(0x7fbd10066a00 msgr2=0x7fbd10145c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:49.245 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:49.265+0000 7fbd15043640 1 -- 172.21.15.114:0/200277826 shutdown_connections 2024-03-29T11:23:49.245 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:49.265+0000 7fbd15043640 1 -- 172.21.15.114:0/200277826 wait complete. 2024-03-29T11:23:49.245 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:49.265+0000 7fbd15043640 1 librados: shutdown 2024-03-29T11:23:49.263 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:23:56.071 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:23:56.219 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.238+0000 7f442ffff640 10 monclient: get_monmap_and_config 2024-03-29T11:23:56.219 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.238+0000 7f442ffff640 10 monclient: build_initial_monmap 2024-03-29T11:23:56.219 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.238+0000 7f442ffff640 10 monclient: monmap: 2024-03-29T11:23:56.220 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:23:56.220 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:23:56.220 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:23:56.239933+0000 2024-03-29T11:23:56.220 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:23:56.239933+0000 2024-03-29T11:23:56.220 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:56.220 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:23:56.221 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:23:56.221 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T11:23:56.221 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T11:23:56.221 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:23:56.224 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.242+0000 7f442ffff640 1 Processor -- start 2024-03-29T11:23:56.224 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.242+0000 7f442ffff640 1 -- start start 2024-03-29T11:23:56.224 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.242+0000 7f442ffff640 10 monclient: init 2024-03-29T11:23:56.224 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.243+0000 7f442ffff640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:56.227 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.243+0000 7f442ffff640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:23:56.227 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.243+0000 7f442ffff640 10 monclient(hunting): picked mon.noname-c con 0x7f44300af390 addr v1:172.21.15.176:6789/0 2024-03-29T11:23:56.227 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.243+0000 7f442ffff640 10 monclient(hunting): picked mon.noname-a con 0x7f44300b3990 addr v1:172.21.15.114:6789/0 2024-03-29T11:23:56.227 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.243+0000 7f442ffff640 10 monclient(hunting): picked mon.noname-b con 0x7f44300b16d0 addr v1:172.21.15.114:6790/0 2024-03-29T11:23:56.227 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.243+0000 7f442ffff640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44300b47f0 con 0x7f44300b3990 2024-03-29T11:23:56.228 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.243+0000 7f442ffff640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44300b1b20 con 0x7f44300af390 2024-03-29T11:23:56.228 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.243+0000 7f442ffff640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44300b44a0 con 0x7f44300b16d0 2024-03-29T11:23:56.228 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.243+0000 7f442ffff640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:56.228 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.243+0000 7f442ffff640 10 monclient(hunting): authenticate will time out at 1106.425976s 2024-03-29T11:23:56.231 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.247+0000 7f442f7fe640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f44300b16d0 0x7f4430151d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:56499/0 (socket says 172.21.15.114:56499) 2024-03-29T11:23:56.231 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.247+0000 7f442f7fe640 1 -- 172.21.15.114:0/3240649350 learned_addr learned my addr 172.21.15.114:0/3240649350 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:23:56.231 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.248+0000 7f442e7fc640 1 -- 172.21.15.114:0/3240649350 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1225112613 0 0) 0x7f44300b44a0 con 0x7f44300b16d0 2024-03-29T11:23:56.231 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.248+0000 7f442e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:23:56.232 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.248+0000 7f442e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:56.232 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.248+0000 7f442e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:56.232 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.248+0000 7f442e7fc640 10 monclient(hunting): my global_id is 4337 2024-03-29T11:23:56.232 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.248+0000 7f442e7fc640 1 -- 172.21.15.114:0/3240649350 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f441c003520 con 0x7f44300b16d0 2024-03-29T11:23:56.255 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.251+0000 7f442e7fc640 1 -- 172.21.15.114:0/3240649350 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2976663057 0 0) 0x7f441c003520 con 0x7f44300b16d0 2024-03-29T11:23:56.255 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.251+0000 7f442e7fc640 1 -- 172.21.15.114:0/3240649350 >> v1:172.21.15.176:6789/0 conn(0x7f44300af390 legacy=0x7f44301459f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:23:56.255 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.251+0000 7f442e7fc640 1 -- 172.21.15.114:0/3240649350 >> v1:172.21.15.114:6789/0 conn(0x7f44300b3990 legacy=0x7f44300b0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:56.255 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.251+0000 7f442e7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:23:56.255 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.251+0000 7f442e7fc640 1 monclient: found mon.noname-b 2024-03-29T11:23:56.255 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.251+0000 7f442e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:56.255 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.251+0000 7f442e7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.114:6790/0 2024-03-29T11:23:56.256 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.251+0000 7f442e7fc640 1 -- 172.21.15.114:0/3240649350 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44300b4c80 con 0x7f44300b16d0 2024-03-29T11:23:56.256 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.251+0000 7f442e7fc640 10 monclient: _finish_auth 0 2024-03-29T11:23:56.256 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.251+0000 7f442e7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:23:56.256 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.251+0000 7f442e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:56.256 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.251+0000 7f442e7fc640 1 -- 172.21.15.114:0/3240649350 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f4428001690 con 0x7f44300b16d0 2024-03-29T11:23:56.256 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.251+0000 7f442e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:56.257 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.251+0000 7f442e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:23:56.257 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.251+0000 7f442e7fc640 10 monclient: dump: 2024-03-29T11:23:56.257 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:23:56.257 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:23:56.257 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:23:56.257 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:23:56.258 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:56.258 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:23:56.258 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:23:56.258 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:23:56.258 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:23:56.258 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:23:56.258 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.251+0000 7f442ffff640 5 monclient: authenticate success, global_id 4337 2024-03-29T11:23:56.259 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.251+0000 7f442ffff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:56.259 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.252+0000 7f442e7fc640 1 -- 172.21.15.114:0/3240649350 <== mon.2 v1:172.21.15.114:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4428001860 con 0x7f44300b16d0 2024-03-29T11:23:56.259 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.252+0000 7f442e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:56.259 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.252+0000 7f442e7fc640 1 -- 172.21.15.114:0/3240649350 <== mon.2 v1:172.21.15.114:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f4428001cb0 con 0x7f44300b16d0 2024-03-29T11:23:56.259 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.252+0000 7f442e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:56.259 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.252+0000 7f442e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:23:56.260 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.252+0000 7f442e7fc640 10 monclient: dump: 2024-03-29T11:23:56.260 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:23:56.260 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:23:56.260 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:23:56.260 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:23:56.260 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:56.261 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:23:56.261 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:23:56.261 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:23:56.261 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:23:56.261 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:23:56.261 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.252+0000 7f442ffff640 10 monclient: get_monmap_and_config success 2024-03-29T11:23:56.262 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.252+0000 7f442ffff640 4 set_mon_vals no callback set 2024-03-29T11:23:56.262 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.252+0000 7f442ffff640 10 monclient: shutdown 2024-03-29T11:23:56.262 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.252+0000 7f442ffff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:56.262 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.252+0000 7f442ffff640 1 -- 172.21.15.114:0/3240649350 >> v1:172.21.15.114:6790/0 conn(0x7f44300b16d0 legacy=0x7f4430151d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:56.262 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.253+0000 7f442ffff640 1 -- 172.21.15.114:0/3240649350 shutdown_connections 2024-03-29T11:23:56.262 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.253+0000 7f442ffff640 1 -- 172.21.15.114:0/3240649350 >> 172.21.15.114:0/3240649350 conn(0x7f443006d230 msgr2=0x7f4430066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:56.263 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.253+0000 7f442ffff640 1 -- 172.21.15.114:0/3240649350 shutdown_connections 2024-03-29T11:23:56.263 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.253+0000 7f442ffff640 1 -- 172.21.15.114:0/3240649350 wait complete. 2024-03-29T11:23:56.263 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.254+0000 7f442ffff640 10 monclient: build_initial_monmap 2024-03-29T11:23:56.263 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.254+0000 7f442ffff640 10 monclient: monmap: 2024-03-29T11:23:56.263 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:23:56.263 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:23:56.263 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T11:23:56.264 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T11:23:56.264 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:56.264 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:23:56.264 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:23:56.264 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T11:23:56.264 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T11:23:56.265 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:23:56.265 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.254+0000 7f442ffff640 1 librados: starting msgr at 2024-03-29T11:23:56.265 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.254+0000 7f442ffff640 1 librados: starting objecter 2024-03-29T11:23:56.265 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.254+0000 7f442ffff640 1 Processor -- start 2024-03-29T11:23:56.265 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.254+0000 7f442ffff640 1 -- start start 2024-03-29T11:23:56.265 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.254+0000 7f442ffff640 1 librados: setting wanted keys 2024-03-29T11:23:56.266 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.254+0000 7f442ffff640 1 librados: calling monclient init 2024-03-29T11:23:56.266 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.254+0000 7f442ffff640 10 monclient: init 2024-03-29T11:23:56.266 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.255+0000 7f442ffff640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:56.266 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.255+0000 7f442ffff640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:23:56.266 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.255+0000 7f442ffff640 10 monclient(hunting): picked mon.noname-a con 0x7f44300af390 addr v1:172.21.15.114:6789/0 2024-03-29T11:23:56.266 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.255+0000 7f442ffff640 10 monclient(hunting): picked mon.noname-c con 0x7f44300b16d0 addr v1:172.21.15.114:6790/0 2024-03-29T11:23:56.267 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.255+0000 7f442ffff640 10 monclient(hunting): picked mon.noname-b con 0x7f44300b3990 addr v1:172.21.15.176:6789/0 2024-03-29T11:23:56.267 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.255+0000 7f442ffff640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44300ae300 con 0x7f44300af390 2024-03-29T11:23:56.267 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.255+0000 7f442ffff640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4430071520 con 0x7f44300b3990 2024-03-29T11:23:56.267 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.255+0000 7f442ffff640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44300716b0 con 0x7f44300b16d0 2024-03-29T11:23:56.267 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.255+0000 7f442ffff640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:56.267 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.255+0000 7f442ffff640 10 monclient(hunting): authenticate will time out at 1106.437976s 2024-03-29T11:23:56.267 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.259+0000 7f44267fc640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f44300b16d0 0x7f44300ad470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:56591/0 (socket says 172.21.15.114:56591) 2024-03-29T11:23:56.268 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.259+0000 7f44267fc640 1 -- 172.21.15.114:0/4066894161 learned_addr learned my addr 172.21.15.114:0/4066894161 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:23:56.268 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.259+0000 7f442cff9640 10 client.?.objecter ms_handle_connect 0x7f44300b16d0 2024-03-29T11:23:56.268 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.259+0000 7f442cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:56.268 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.260+0000 7f442cff9640 1 -- 172.21.15.114:0/4066894161 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 454465092 0 0) 0x7f44300716b0 con 0x7f44300b16d0 2024-03-29T11:23:56.268 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.260+0000 7f442cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:23:56.268 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.260+0000 7f442cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:56.269 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.260+0000 7f442cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:56.269 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.260+0000 7f442cff9640 10 monclient(hunting): my global_id is 4340 2024-03-29T11:23:56.269 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.260+0000 7f442cff9640 1 -- 172.21.15.114:0/4066894161 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4410004eb0 con 0x7f44300b16d0 2024-03-29T11:23:56.269 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.262+0000 7f442cff9640 1 -- 172.21.15.114:0/4066894161 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2238274731 0 0) 0x7f4410004eb0 con 0x7f44300b16d0 2024-03-29T11:23:56.269 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.262+0000 7f442cff9640 1 -- 172.21.15.114:0/4066894161 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4410005300 con 0x7f44300b16d0 2024-03-29T11:23:56.269 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.262+0000 7f442cff9640 1 -- 172.21.15.114:0/4066894161 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f4420001c70 con 0x7f44300b16d0 2024-03-29T11:23:56.270 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.262+0000 7f442cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:56.270 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.262+0000 7f442cff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:23:56.270 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.262+0000 7f442cff9640 10 monclient(hunting): dump: 2024-03-29T11:23:56.270 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:23:56.270 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:23:56.270 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:23:56.270 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:23:56.271 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:56.271 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:23:56.271 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:23:56.271 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:23:56.271 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:23:56.271 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:23:56.272 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.262+0000 7f442cff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:23:56.272 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.262+0000 7f442cff9640 10 client.?.objecter ms_handle_connect 0x7f44300af390 2024-03-29T11:23:56.272 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.262+0000 7f442cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:56.272 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.263+0000 7f442cff9640 1 -- 172.21.15.114:0/4066894161 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3412602963 0 0) 0x7f44300ae300 con 0x7f44300af390 2024-03-29T11:23:56.272 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.263+0000 7f442cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:23:56.272 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.263+0000 7f442cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:56.272 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.263+0000 7f442cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:56.273 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.263+0000 7f442cff9640 10 monclient(hunting): my global_id is 4338 2024-03-29T11:23:56.273 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.263+0000 7f442cff9640 1 -- 172.21.15.114:0/4066894161 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4410004eb0 con 0x7f44300af390 2024-03-29T11:23:56.273 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.263+0000 7f442cff9640 1 -- 172.21.15.114:0/4066894161 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4230630079 0 0) 0x7f4410004eb0 con 0x7f44300af390 2024-03-29T11:23:56.273 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.263+0000 7f442cff9640 1 -- 172.21.15.114:0/4066894161 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4410005bb0 con 0x7f44300af390 2024-03-29T11:23:56.273 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.263+0000 7f442cff9640 1 -- 172.21.15.114:0/4066894161 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f4418001750 con 0x7f44300af390 2024-03-29T11:23:56.273 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.263+0000 7f442cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:56.274 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.263+0000 7f442cff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:23:56.274 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.263+0000 7f442cff9640 10 monclient(hunting): dump: 2024-03-29T11:23:56.274 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:23:56.274 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:23:56.274 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:23:56.274 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:23:56.274 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:56.275 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:23:56.275 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:23:56.275 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:23:56.275 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:23:56.275 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:23:56.275 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.264+0000 7f442cff9640 1 -- 172.21.15.114:0/4066894161 <== mon.0 v1:172.21.15.114:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2521589811 0 0) 0x7f4410005bb0 con 0x7f44300af390 2024-03-29T11:23:56.276 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.264+0000 7f442cff9640 1 -- 172.21.15.114:0/4066894161 >> v1:172.21.15.114:6790/0 conn(0x7f44300b16d0 legacy=0x7f44300ad470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:56.276 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.264+0000 7f442cff9640 1 -- 172.21.15.114:0/4066894161 >> v1:172.21.15.176:6789/0 conn(0x7f44300b3990 legacy=0x7f44301de650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:56.276 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.264+0000 7f442cff9640 10 monclient: _finish_hunting 0 2024-03-29T11:23:56.276 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.264+0000 7f442cff9640 1 monclient: found mon.a 2024-03-29T11:23:56.276 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.264+0000 7f442cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:56.276 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.264+0000 7f442cff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T11:23:56.276 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.264+0000 7f442cff9640 1 -- 172.21.15.114:0/4066894161 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4430070070 con 0x7f44300af390 2024-03-29T11:23:56.277 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.264+0000 7f442cff9640 10 monclient: _finish_auth 0 2024-03-29T11:23:56.277 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.264+0000 7f442cff9640 10 monclient: _check_auth_tickets 2024-03-29T11:23:56.277 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.264+0000 7f442cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:56.277 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.264+0000 7f442ffff640 5 monclient: authenticate success, global_id 4338 2024-03-29T11:23:56.277 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.264+0000 7f442ffff640 10 monclient: _renew_subs 2024-03-29T11:23:56.277 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.264+0000 7f442ffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T11:23:56.278 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.265+0000 7f442ffff640 1 -- 172.21.15.114:0/4066894161 --> v1:172.21.15.114:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4430070320 con 0x7f44300af390 2024-03-29T11:23:56.278 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.265+0000 7f442ffff640 10 client.4338.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:23:56.278 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.265+0000 7f442ffff640 10 monclient: _renew_subs 2024-03-29T11:23:56.278 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.265+0000 7f442ffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T11:23:56.278 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.265+0000 7f442ffff640 1 -- 172.21.15.114:0/4066894161 --> v1:172.21.15.114:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f44300add60 con 0x7f44300af390 2024-03-29T11:23:56.278 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.265+0000 7f442ffff640 1 librados: init done 2024-03-29T11:23:56.279 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.266+0000 7f442ffff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:56.279 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.267+0000 7f442ffff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:56.279 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.267+0000 7f442ffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T11:23:56.279 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.267+0000 7f442ffff640 1 -- 172.21.15.114:0/4066894161 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44300ae060 con 0x7f44300af390 2024-03-29T11:23:56.279 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.269+0000 7f442cff9640 1 -- 172.21.15.114:0/4066894161 <== mon.0 v1:172.21.15.114:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4418001cb0 con 0x7f44300af390 2024-03-29T11:23:56.279 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.269+0000 7f442cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:56.280 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.269+0000 7f442cff9640 1 -- 172.21.15.114:0/4066894161 <== mon.0 v1:172.21.15.114:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f4418001750 con 0x7f44300af390 2024-03-29T11:23:56.280 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.269+0000 7f442cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:56.280 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.269+0000 7f442cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:23:56.280 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.269+0000 7f442cff9640 10 monclient: dump: 2024-03-29T11:23:56.280 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:23:56.280 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:23:56.281 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:23:56.281 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:23:56.281 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:56.281 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:23:56.281 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:23:56.282 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:23:56.282 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:23:56.282 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:23:56.282 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.269+0000 7f442cff9640 1 -- 172.21.15.114:0/4066894161 <== mon.0 v1:172.21.15.114:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1594292461 0 0) 0x7f4418019460 con 0x7f44300af390 2024-03-29T11:23:56.282 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.270+0000 7f442cff9640 1 -- 172.21.15.114:0/4066894161 <== mon.0 v1:172.21.15.114:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3238987313 0 0) 0x7f4418083450 con 0x7f44300af390 2024-03-29T11:23:56.283 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.270+0000 7f442cff9640 10 client.4338.objecter ms_dispatch 0x7f4430000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:23:56.283 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.270+0000 7f442cff9640 3 client.4338.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:23:56.283 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.270+0000 7f442cff9640 3 client.4338.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:23:56.283 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.270+0000 7f442cff9640 20 client.4338.objecter dump_active .. 0 homeless 2024-03-29T11:23:56.284 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.270+0000 7f44267fc640 10 monclient: get_auth_request con 0x7f441006e600 auth_method 0 2024-03-29T11:23:56.284 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.270+0000 7f442cff9640 10 client.4338.objecter ms_handle_connect 0x7f441006e600 2024-03-29T11:23:56.284 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.272+0000 7f442dffb640 4 set_mon_vals no callback set 2024-03-29T11:23:56.285 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.278+0000 7f442cff9640 1 -- 172.21.15.114:0/4066894161 <== mon.0 v1:172.21.15.114:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f441804ebe0 con 0x7f44300af390 2024-03-29T11:23:56.285 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.278+0000 7f442cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:56.286 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.278+0000 7f442cff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:23:56.590 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.610+0000 7f442ffff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:56.591 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.610+0000 7f442ffff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:56.591 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.610+0000 7f442ffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T11:23:56.591 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.610+0000 7f442ffff640 1 -- 172.21.15.114:0/4066894161 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f443004f730 con 0x7f44300af390 2024-03-29T11:23:56.591 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.610+0000 7f442cff9640 1 -- 172.21.15.114:0/4066894161 <== mon.0 v1:172.21.15.114:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2858224613) 0x7f441804ebe0 con 0x7f44300af390 2024-03-29T11:23:56.592 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.610+0000 7f442cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:56.592 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.610+0000 7f442cff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:23:56.595 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T11:23:56.595 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"b0de2b9b-c45a-44c7-b6ee-e83b39a1e108","created":"2024-03-29T11:20:49.420660+0000","modified":"2024-03-29T11:20:51.700167+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:51.700167+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a64785d3-6b11-499a-b706-cc827ab96a74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5882dadb-e7a8-4604-ba9a-418013c106c8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d7e8c409-bc66-4e4a-bbdb-420bad31388e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0682989-3595-4cd6-90a5-ab9111c278e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"67068120-e3ff-4817-bd06-44feb7479a35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09e8b-8a85-4997-bdea-75d754e6ee67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2f0d9ead-f47f-4cb3-92a4-0a5ad23cd831","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fdb6a25f-6bd8-4c65-9b9b-b460ba7673a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:23:56.598 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.618+0000 7f442ffff640 10 librados: watch_flush enter 2024-03-29T11:23:56.598 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.618+0000 7f442ffff640 10 librados: watch_flush exit 2024-03-29T11:23:56.599 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.618+0000 7f442ffff640 20 client.4338.objecter shutdown clearing up homeless session... 2024-03-29T11:23:56.599 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.618+0000 7f442ffff640 10 client.4338.objecter successfully canceled tick 2024-03-29T11:23:56.599 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.618+0000 7f442ffff640 1 -- 172.21.15.114:0/4066894161 >> v1:172.21.15.114:6816/4038755055 conn(0x7f441006e600 legacy=0x7f4410070ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:56.599 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.618+0000 7f442ffff640 10 monclient: shutdown 2024-03-29T11:23:56.599 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.618+0000 7f442ffff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:56.599 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.618+0000 7f442ffff640 1 -- 172.21.15.114:0/4066894161 >> v1:172.21.15.114:6789/0 conn(0x7f44300af390 legacy=0x7f4430070e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:56.599 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.618+0000 7f442ffff640 1 -- 172.21.15.114:0/4066894161 shutdown_connections 2024-03-29T11:23:56.599 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.618+0000 7f442ffff640 1 -- 172.21.15.114:0/4066894161 >> 172.21.15.114:0/4066894161 conn(0x7f443006d230 msgr2=0x7f44300b7220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:56.599 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.619+0000 7f442ffff640 1 -- 172.21.15.114:0/4066894161 shutdown_connections 2024-03-29T11:23:56.600 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.619+0000 7f442ffff640 1 -- 172.21.15.114:0/4066894161 wait complete. 2024-03-29T11:23:56.602 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:23:56.621+0000 7f442ffff640 1 librados: shutdown 2024-03-29T11:23:56.619 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:24:03.426 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:24:03.592 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.611+0000 7f3866952640 10 monclient: get_monmap_and_config 2024-03-29T11:24:03.592 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.611+0000 7f3866952640 10 monclient: build_initial_monmap 2024-03-29T11:24:03.592 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.611+0000 7f3866952640 10 monclient: monmap: 2024-03-29T11:24:03.592 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:24:03.592 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:24:03.592 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:24:03.612798+0000 2024-03-29T11:24:03.593 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:24:03.612798+0000 2024-03-29T11:24:03.593 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:03.593 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:24:03.593 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:24:03.593 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T11:24:03.593 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T11:24:03.593 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:24:03.594 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.613+0000 7f3866952640 1 Processor -- start 2024-03-29T11:24:03.594 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.613+0000 7f3866952640 1 -- start start 2024-03-29T11:24:03.594 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.613+0000 7f3866952640 10 monclient: init 2024-03-29T11:24:03.596 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.615+0000 7f3866952640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:03.596 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.615+0000 7f3866952640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:24:03.596 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.615+0000 7f3866952640 10 monclient(hunting): picked mon.noname-c con 0x7f38601457f0 addr v1:172.21.15.176:6789/0 2024-03-29T11:24:03.596 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.615+0000 7f3866952640 10 monclient(hunting): picked mon.noname-a con 0x7f386013f650 addr v1:172.21.15.114:6789/0 2024-03-29T11:24:03.596 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.615+0000 7f3866952640 10 monclient(hunting): picked mon.noname-b con 0x7f38600a9d70 addr v1:172.21.15.114:6790/0 2024-03-29T11:24:03.596 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.615+0000 7f3866952640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38600aaef0 con 0x7f386013f650 2024-03-29T11:24:03.596 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.615+0000 7f3866952640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3860145bd0 con 0x7f38601457f0 2024-03-29T11:24:03.597 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.615+0000 7f3866952640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3860145eb0 con 0x7f38600a9d70 2024-03-29T11:24:03.597 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.615+0000 7f3866952640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:03.597 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.615+0000 7f3866952640 10 monclient(hunting): authenticate will time out at 1113.797310s 2024-03-29T11:24:03.598 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.617+0000 7f3864eef640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f38600a9d70 0x7f386014d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:39882/0 (socket says 172.21.15.114:39882) 2024-03-29T11:24:03.598 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.617+0000 7f3864eef640 1 -- 172.21.15.114:0/2920601509 learned_addr learned my addr 172.21.15.114:0/2920601509 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:24:03.600 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.618+0000 7f385effd640 1 -- 172.21.15.114:0/2920601509 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1210137717 0 0) 0x7f3860145eb0 con 0x7f38600a9d70 2024-03-29T11:24:03.600 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.618+0000 7f385effd640 10 monclient(hunting): _negotiate 2024-03-29T11:24:03.600 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.618+0000 7f385effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:03.600 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.618+0000 7f385effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:03.600 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.618+0000 7f385effd640 10 monclient(hunting): my global_id is 4343 2024-03-29T11:24:03.600 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.618+0000 7f385effd640 1 -- 172.21.15.114:0/2920601509 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3848003520 con 0x7f38600a9d70 2024-03-29T11:24:03.600 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.619+0000 7f385effd640 1 -- 172.21.15.114:0/2920601509 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 487169823 0 0) 0x7f3848003520 con 0x7f38600a9d70 2024-03-29T11:24:03.601 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.619+0000 7f385effd640 1 -- 172.21.15.114:0/2920601509 >> v1:172.21.15.176:6789/0 conn(0x7f38601457f0 legacy=0x7f386013ef40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T11:24:03.601 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.619+0000 7f385effd640 1 -- 172.21.15.114:0/2920601509 >> v1:172.21.15.114:6789/0 conn(0x7f386013f650 legacy=0x7f38600b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:24:03.602 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.621+0000 7f385effd640 10 monclient: _finish_hunting 0 2024-03-29T11:24:03.602 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.621+0000 7f385effd640 1 monclient: found mon.noname-b 2024-03-29T11:24:03.602 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.621+0000 7f385effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:03.602 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.621+0000 7f385effd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.114:6790/0 2024-03-29T11:24:03.602 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.621+0000 7f385effd640 1 -- 172.21.15.114:0/2920601509 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3860146340 con 0x7f38600a9d70 2024-03-29T11:24:03.603 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.621+0000 7f385effd640 10 monclient: _finish_auth 0 2024-03-29T11:24:03.603 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.621+0000 7f385effd640 10 monclient: _check_auth_tickets 2024-03-29T11:24:03.603 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.621+0000 7f385effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:03.603 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.621+0000 7f3866952640 5 monclient: authenticate success, global_id 4343 2024-03-29T11:24:03.603 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.621+0000 7f3866952640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:03.603 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.621+0000 7f385effd640 1 -- 172.21.15.114:0/2920601509 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f3858001a10 con 0x7f38600a9d70 2024-03-29T11:24:03.603 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.621+0000 7f385effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:03.603 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.621+0000 7f385effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:24:03.603 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.621+0000 7f385effd640 10 monclient: dump: 2024-03-29T11:24:03.603 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:24:03.603 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:24:03.603 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:24:03.603 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:24:03.603 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:03.604 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:24:03.604 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:24:03.604 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:24:03.604 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:24:03.604 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:24:03.604 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.621+0000 7f3866952640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:03.605 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.624+0000 7f385effd640 1 -- 172.21.15.114:0/2920601509 <== mon.2 v1:172.21.15.114:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3858001be0 con 0x7f38600a9d70 2024-03-29T11:24:03.606 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.624+0000 7f385effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:03.606 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.624+0000 7f385effd640 1 -- 172.21.15.114:0/2920601509 <== mon.2 v1:172.21.15.114:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f3858002030 con 0x7f38600a9d70 2024-03-29T11:24:03.606 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.624+0000 7f385effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:03.606 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.624+0000 7f385effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:03.606 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.624+0000 7f385effd640 10 monclient: dump: 2024-03-29T11:24:03.606 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:24:03.606 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:24:03.606 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:24:03.606 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:24:03.606 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:03.606 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:24:03.606 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:24:03.606 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:24:03.606 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:24:03.607 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:24:03.607 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.625+0000 7f3866952640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:03.607 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.625+0000 7f3866952640 4 set_mon_vals no callback set 2024-03-29T11:24:03.607 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.626+0000 7f3866952640 10 monclient: shutdown 2024-03-29T11:24:03.607 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.626+0000 7f3866952640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:03.608 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.626+0000 7f3866952640 1 -- 172.21.15.114:0/2920601509 >> v1:172.21.15.114:6790/0 conn(0x7f38600a9d70 legacy=0x7f386014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:03.608 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.626+0000 7f3866952640 1 -- 172.21.15.114:0/2920601509 shutdown_connections 2024-03-29T11:24:03.608 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.626+0000 7f3866952640 1 -- 172.21.15.114:0/2920601509 >> 172.21.15.114:0/2920601509 conn(0x7f3860066a00 msgr2=0x7f3860066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:03.608 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.627+0000 7f3866952640 1 -- 172.21.15.114:0/2920601509 shutdown_connections 2024-03-29T11:24:03.610 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.627+0000 7f3866952640 1 -- 172.21.15.114:0/2920601509 wait complete. 2024-03-29T11:24:03.610 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.628+0000 7f3866952640 10 monclient: build_initial_monmap 2024-03-29T11:24:03.610 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.628+0000 7f3866952640 10 monclient: monmap: 2024-03-29T11:24:03.610 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:24:03.610 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:03.610 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T11:24:03.610 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T11:24:03.611 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:03.611 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:24:03.611 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:24:03.611 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T11:24:03.611 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T11:24:03.611 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:24:03.611 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.628+0000 7f3866952640 1 librados: starting msgr at 2024-03-29T11:24:03.611 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.628+0000 7f3866952640 1 librados: starting objecter 2024-03-29T11:24:03.611 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.629+0000 7f3866952640 1 Processor -- start 2024-03-29T11:24:03.611 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.629+0000 7f3866952640 1 -- start start 2024-03-29T11:24:03.611 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.629+0000 7f3866952640 1 librados: setting wanted keys 2024-03-29T11:24:03.611 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.629+0000 7f3866952640 1 librados: calling monclient init 2024-03-29T11:24:03.611 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.629+0000 7f3866952640 10 monclient: init 2024-03-29T11:24:03.613 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.630+0000 7f3866952640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:03.613 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.630+0000 7f3866952640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:24:03.613 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.630+0000 7f3866952640 10 monclient(hunting): picked mon.noname-c con 0x7f38600a9d70 addr v1:172.21.15.114:6790/0 2024-03-29T11:24:03.613 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.630+0000 7f3866952640 10 monclient(hunting): picked mon.noname-b con 0x7f386013f650 addr v1:172.21.15.176:6789/0 2024-03-29T11:24:03.613 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.630+0000 7f3866952640 10 monclient(hunting): picked mon.noname-a con 0x7f38601457f0 addr v1:172.21.15.114:6789/0 2024-03-29T11:24:03.613 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.630+0000 7f3866952640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38600ab830 con 0x7f38601457f0 2024-03-29T11:24:03.613 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.630+0000 7f3866952640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3860070e30 con 0x7f386013f650 2024-03-29T11:24:03.613 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.630+0000 7f3866952640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38600b15e0 con 0x7f38600a9d70 2024-03-29T11:24:03.613 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.630+0000 7f3866952640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:03.613 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.630+0000 7f3866952640 10 monclient(hunting): authenticate will time out at 1113.812544s 2024-03-29T11:24:03.613 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.630+0000 7f385ffff640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f38600a9d70 0x7f3860070050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:49980/0 (socket says 172.21.15.114:49980) 2024-03-29T11:24:03.614 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.630+0000 7f385ffff640 1 -- 172.21.15.114:0/357346989 learned_addr learned my addr 172.21.15.114:0/357346989 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:24:03.616 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.635+0000 7f385cff9640 10 client.?.objecter ms_handle_connect 0x7f38600a9d70 2024-03-29T11:24:03.616 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.635+0000 7f385cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:03.621 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.638+0000 7f385cff9640 10 client.?.objecter ms_handle_connect 0x7f38601457f0 2024-03-29T11:24:03.621 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.638+0000 7f385cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:03.621 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.638+0000 7f385cff9640 10 client.?.objecter ms_handle_connect 0x7f386013f650 2024-03-29T11:24:03.621 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.638+0000 7f385cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:03.621 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.638+0000 7f385cff9640 1 -- 172.21.15.114:0/357346989 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 300310746 0 0) 0x7f38600ab830 con 0x7f38601457f0 2024-03-29T11:24:03.621 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.638+0000 7f385cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:24:03.621 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.638+0000 7f385cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:03.622 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.638+0000 7f385cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:03.622 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.638+0000 7f385cff9640 10 monclient(hunting): my global_id is 4341 2024-03-29T11:24:03.622 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.638+0000 7f385cff9640 1 -- 172.21.15.114:0/357346989 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f384c004eb0 con 0x7f38601457f0 2024-03-29T11:24:03.622 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.639+0000 7f385cff9640 1 -- 172.21.15.114:0/357346989 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3208132256 0 0) 0x7f38600b15e0 con 0x7f38600a9d70 2024-03-29T11:24:03.622 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.639+0000 7f385cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:24:03.622 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.639+0000 7f385cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:03.622 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.639+0000 7f385cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:03.622 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.639+0000 7f385cff9640 10 monclient(hunting): my global_id is 4346 2024-03-29T11:24:03.622 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.639+0000 7f385cff9640 1 -- 172.21.15.114:0/357346989 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38600ab830 con 0x7f38600a9d70 2024-03-29T11:24:03.622 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.639+0000 7f385cff9640 1 -- 172.21.15.114:0/357346989 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 332499559 0 0) 0x7f3860070e30 con 0x7f386013f650 2024-03-29T11:24:03.622 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.639+0000 7f385cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:24:03.622 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.639+0000 7f385cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:03.622 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.639+0000 7f385cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:03.623 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.640+0000 7f385cff9640 10 monclient(hunting): my global_id is 4339 2024-03-29T11:24:03.623 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.640+0000 7f385cff9640 1 -- 172.21.15.114:0/357346989 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38600b15e0 con 0x7f386013f650 2024-03-29T11:24:03.623 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.640+0000 7f385cff9640 1 -- 172.21.15.114:0/357346989 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1271904827 0 0) 0x7f38600ab830 con 0x7f38600a9d70 2024-03-29T11:24:03.623 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.640+0000 7f385cff9640 1 -- 172.21.15.114:0/357346989 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3860070e30 con 0x7f38600a9d70 2024-03-29T11:24:03.623 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.640+0000 7f385cff9640 1 -- 172.21.15.114:0/357346989 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f38540016d0 con 0x7f38600a9d70 2024-03-29T11:24:03.623 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.640+0000 7f385cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:03.623 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.640+0000 7f385cff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:24:03.623 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.640+0000 7f385cff9640 10 monclient(hunting): dump: 2024-03-29T11:24:03.623 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:24:03.623 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:24:03.624 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:24:03.624 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:24:03.624 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:03.624 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:24:03.624 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:24:03.624 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:24:03.624 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:24:03.624 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:24:03.624 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.640+0000 7f385cff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:03.625 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.640+0000 7f385cff9640 1 -- 172.21.15.114:0/357346989 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2289001587 0 0) 0x7f38600b15e0 con 0x7f386013f650 2024-03-29T11:24:03.625 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.640+0000 7f385cff9640 1 -- 172.21.15.114:0/357346989 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f38600ab830 con 0x7f386013f650 2024-03-29T11:24:03.625 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.640+0000 7f385cff9640 1 -- 172.21.15.114:0/357346989 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f3850001660 con 0x7f386013f650 2024-03-29T11:24:03.626 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.640+0000 7f385cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:03.626 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.640+0000 7f385cff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:24:03.626 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.640+0000 7f385cff9640 10 monclient(hunting): dump: 2024-03-29T11:24:03.626 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:24:03.626 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:24:03.626 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:24:03.626 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:24:03.626 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:03.626 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:24:03.626 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:24:03.627 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:24:03.627 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:24:03.627 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:24:03.627 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.641+0000 7f385cff9640 1 -- 172.21.15.114:0/357346989 <== mon.1 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1797860480 0 0) 0x7f38600ab830 con 0x7f386013f650 2024-03-29T11:24:03.627 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.641+0000 7f385cff9640 1 -- 172.21.15.114:0/357346989 >> v1:172.21.15.114:6790/0 conn(0x7f38600a9d70 legacy=0x7f3860070050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:03.627 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.641+0000 7f385cff9640 1 -- 172.21.15.114:0/357346989 >> v1:172.21.15.114:6789/0 conn(0x7f38601457f0 legacy=0x7f38601eea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:03.627 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.641+0000 7f385cff9640 10 monclient: _finish_hunting 0 2024-03-29T11:24:03.627 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.641+0000 7f385cff9640 1 monclient: found mon.b 2024-03-29T11:24:03.627 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.641+0000 7f385cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:03.627 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.641+0000 7f385cff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:24:03.627 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.641+0000 7f385cff9640 1 -- 172.21.15.114:0/357346989 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3860071350 con 0x7f386013f650 2024-03-29T11:24:03.628 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.641+0000 7f385cff9640 10 monclient: _finish_auth 0 2024-03-29T11:24:03.628 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.641+0000 7f385cff9640 10 monclient: _check_auth_tickets 2024-03-29T11:24:03.628 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.641+0000 7f385cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:03.628 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.641+0000 7f3866952640 5 monclient: authenticate success, global_id 4339 2024-03-29T11:24:03.628 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.641+0000 7f3866952640 10 monclient: _renew_subs 2024-03-29T11:24:03.628 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.641+0000 7f3866952640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:24:03.628 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.641+0000 7f3866952640 1 -- 172.21.15.114:0/357346989 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3860070880 con 0x7f386013f650 2024-03-29T11:24:03.628 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.641+0000 7f385cff9640 1 -- 172.21.15.114:0/357346989 <== mon.1 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3850001b70 con 0x7f386013f650 2024-03-29T11:24:03.628 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.641+0000 7f385cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:03.628 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.641+0000 7f385cff9640 1 -- 172.21.15.114:0/357346989 <== mon.1 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f3850003790 con 0x7f386013f650 2024-03-29T11:24:03.629 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.641+0000 7f385cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:03.629 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.641+0000 7f385cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:24:03.629 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.641+0000 7f385cff9640 10 monclient: dump: 2024-03-29T11:24:03.629 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:24:03.629 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:24:03.629 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:24:03.629 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:24:03.629 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:03.629 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:24:03.630 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:24:03.630 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:24:03.630 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:24:03.630 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:24:03.630 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.641+0000 7f3866952640 10 client.4339.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:03.630 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.641+0000 7f3866952640 10 monclient: _renew_subs 2024-03-29T11:24:03.630 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.641+0000 7f3866952640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:24:03.630 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.641+0000 7f3866952640 1 -- 172.21.15.114:0/357346989 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f38601e4910 con 0x7f386013f650 2024-03-29T11:24:03.630 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.641+0000 7f3866952640 1 librados: init done 2024-03-29T11:24:03.630 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.641+0000 7f385e7fc640 4 set_mon_vals no callback set 2024-03-29T11:24:03.630 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.644+0000 7f3866952640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:03.630 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.644+0000 7f3866952640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:03.630 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.644+0000 7f3866952640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:24:03.630 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.644+0000 7f3866952640 1 -- 172.21.15.114:0/357346989 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3860070fc0 con 0x7f386013f650 2024-03-29T11:24:03.631 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.645+0000 7f385cff9640 1 -- 172.21.15.114:0/357346989 <== mon.1 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1594292461 0 0) 0x7f3850019760 con 0x7f386013f650 2024-03-29T11:24:03.632 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.646+0000 7f385cff9640 1 -- 172.21.15.114:0/357346989 <== mon.1 v1:172.21.15.176:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3238987313 0 0) 0x7f3850083670 con 0x7f386013f650 2024-03-29T11:24:03.632 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.646+0000 7f385cff9640 10 client.4339.objecter ms_dispatch 0x7f3860000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:24:03.632 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.646+0000 7f385cff9640 3 client.4339.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:24:03.632 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.646+0000 7f385ffff640 10 monclient: get_auth_request con 0x7f384c06dfd0 auth_method 0 2024-03-29T11:24:03.632 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.647+0000 7f385cff9640 3 client.4339.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:24:03.632 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.647+0000 7f385cff9640 20 client.4339.objecter dump_active .. 0 homeless 2024-03-29T11:24:03.632 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.647+0000 7f385cff9640 10 client.4339.objecter ms_handle_connect 0x7f384c06dfd0 2024-03-29T11:24:03.632 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.650+0000 7f385cff9640 1 -- 172.21.15.114:0/357346989 <== mon.1 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f385004ef50 con 0x7f386013f650 2024-03-29T11:24:03.632 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.650+0000 7f385cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:03.632 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:03.651+0000 7f385cff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:04.008 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:04.026+0000 7f3866952640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:04.009 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:04.026+0000 7f3866952640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:04.009 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:04.026+0000 7f3866952640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:24:04.009 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:04.026+0000 7f3866952640 1 -- 172.21.15.114:0/357346989 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f38601452e0 con 0x7f386013f650 2024-03-29T11:24:04.010 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:04.030+0000 7f385cff9640 1 -- 172.21.15.114:0/357346989 <== mon.1 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2858224613) 0x7f3850050e80 con 0x7f386013f650 2024-03-29T11:24:04.011 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:04.030+0000 7f385cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:04.011 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:04.030+0000 7f385cff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:04.011 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T11:24:04.011 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"b0de2b9b-c45a-44c7-b6ee-e83b39a1e108","created":"2024-03-29T11:20:49.420660+0000","modified":"2024-03-29T11:20:51.700167+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:51.700167+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a64785d3-6b11-499a-b706-cc827ab96a74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5882dadb-e7a8-4604-ba9a-418013c106c8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d7e8c409-bc66-4e4a-bbdb-420bad31388e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0682989-3595-4cd6-90a5-ab9111c278e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"67068120-e3ff-4817-bd06-44feb7479a35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09e8b-8a85-4997-bdea-75d754e6ee67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2f0d9ead-f47f-4cb3-92a4-0a5ad23cd831","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fdb6a25f-6bd8-4c65-9b9b-b460ba7673a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:24:04.015 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:04.034+0000 7f3866952640 10 librados: watch_flush enter 2024-03-29T11:24:04.016 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:04.034+0000 7f3866952640 10 librados: watch_flush exit 2024-03-29T11:24:04.016 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:04.034+0000 7f3866952640 20 client.4339.objecter shutdown clearing up homeless session... 2024-03-29T11:24:04.016 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:04.034+0000 7f3866952640 10 client.4339.objecter successfully canceled tick 2024-03-29T11:24:04.016 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:04.035+0000 7f3866952640 1 -- 172.21.15.114:0/357346989 >> v1:172.21.15.114:6816/4038755055 conn(0x7f384c06dfd0 legacy=0x7f384c070490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:04.016 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:04.035+0000 7f3866952640 10 monclient: shutdown 2024-03-29T11:24:04.016 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:04.035+0000 7f3866952640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:04.016 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:04.035+0000 7f3866952640 1 -- 172.21.15.114:0/357346989 >> v1:172.21.15.176:6789/0 conn(0x7f386013f650 legacy=0x7f38601e4020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:04.017 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:04.036+0000 7f3866952640 1 -- 172.21.15.114:0/357346989 shutdown_connections 2024-03-29T11:24:04.017 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:04.036+0000 7f3866952640 1 -- 172.21.15.114:0/357346989 >> 172.21.15.114:0/357346989 conn(0x7f3860066a00 msgr2=0x7f38600b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:04.018 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:04.037+0000 7f3866952640 1 -- 172.21.15.114:0/357346989 shutdown_connections 2024-03-29T11:24:04.018 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:04.037+0000 7f3866952640 1 -- 172.21.15.114:0/357346989 wait complete. 2024-03-29T11:24:04.018 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:04.038+0000 7f3866952640 1 librados: shutdown 2024-03-29T11:24:04.038 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:24:10.845 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:24:10.998 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.017+0000 7f45d2b3d640 10 monclient: get_monmap_and_config 2024-03-29T11:24:10.998 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.017+0000 7f45d2b3d640 10 monclient: build_initial_monmap 2024-03-29T11:24:10.998 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.017+0000 7f45d2b3d640 10 monclient: monmap: 2024-03-29T11:24:10.999 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:24:10.999 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:24:10.999 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:24:11.018826+0000 2024-03-29T11:24:10.999 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:24:11.018826+0000 2024-03-29T11:24:10.999 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:10.999 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:24:10.999 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:24:10.999 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T11:24:10.999 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T11:24:10.999 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:24:11.000 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.019+0000 7f45d2b3d640 1 Processor -- start 2024-03-29T11:24:11.000 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.019+0000 7f45d2b3d640 1 -- start start 2024-03-29T11:24:11.000 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.019+0000 7f45d2b3d640 10 monclient: init 2024-03-29T11:24:11.001 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.020+0000 7f45d2b3d640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:11.001 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.020+0000 7f45d2b3d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:24:11.002 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.020+0000 7f45d2b3d640 10 monclient(hunting): picked mon.noname-a con 0x7f45cc0b4330 addr v1:172.21.15.114:6789/0 2024-03-29T11:24:11.002 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.020+0000 7f45d2b3d640 10 monclient(hunting): picked mon.noname-c con 0x7f45cc1470b0 addr v1:172.21.15.176:6789/0 2024-03-29T11:24:11.002 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.020+0000 7f45d2b3d640 10 monclient(hunting): picked mon.noname-b con 0x7f45cc1439b0 addr v1:172.21.15.114:6790/0 2024-03-29T11:24:11.002 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.020+0000 7f45d2b3d640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45cc06f2a0 con 0x7f45cc0b4330 2024-03-29T11:24:11.002 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.020+0000 7f45d2b3d640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45cc146b90 con 0x7f45cc1470b0 2024-03-29T11:24:11.002 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.020+0000 7f45d2b3d640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45cc0b2ca0 con 0x7f45cc1439b0 2024-03-29T11:24:11.002 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.020+0000 7f45d2b3d640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:11.002 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.020+0000 7f45d2b3d640 10 monclient(hunting): authenticate will time out at 1121.202407s 2024-03-29T11:24:11.002 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.020+0000 7f45d10da640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f45cc1439b0 0x7f45cc151d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:42466/0 (socket says 172.21.15.114:42466) 2024-03-29T11:24:11.002 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.020+0000 7f45d10da640 1 -- 172.21.15.114:0/752940550 learned_addr learned my addr 172.21.15.114:0/752940550 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:24:11.003 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.021+0000 7f45cb7fe640 1 -- 172.21.15.114:0/752940550 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1047928344 0 0) 0x7f45cc146b90 con 0x7f45cc1470b0 2024-03-29T11:24:11.003 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.021+0000 7f45cb7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:24:11.004 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.021+0000 7f45cb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:11.004 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.021+0000 7f45cb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:11.004 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.021+0000 7f45cb7fe640 10 monclient(hunting): my global_id is 4342 2024-03-29T11:24:11.004 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.021+0000 7f45cb7fe640 1 -- 172.21.15.114:0/752940550 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45b4003520 con 0x7f45cc1470b0 2024-03-29T11:24:11.004 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.021+0000 7f45cb7fe640 1 -- 172.21.15.114:0/752940550 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2752877549 0 0) 0x7f45cc0b2ca0 con 0x7f45cc1439b0 2024-03-29T11:24:11.004 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.021+0000 7f45cb7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:24:11.004 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.021+0000 7f45cb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:11.004 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.021+0000 7f45cb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:11.004 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.021+0000 7f45cb7fe640 10 monclient(hunting): my global_id is 4349 2024-03-29T11:24:11.004 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.021+0000 7f45cb7fe640 1 -- 172.21.15.114:0/752940550 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45b4003840 con 0x7f45cc1439b0 2024-03-29T11:24:11.004 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.021+0000 7f45cb7fe640 1 -- 172.21.15.114:0/752940550 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2668774913 0 0) 0x7f45b4003520 con 0x7f45cc1470b0 2024-03-29T11:24:11.005 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.021+0000 7f45cb7fe640 1 -- 172.21.15.114:0/752940550 >> v1:172.21.15.114:6790/0 conn(0x7f45cc1439b0 legacy=0x7f45cc151d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:11.005 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.021+0000 7f45cb7fe640 1 -- 172.21.15.114:0/752940550 >> v1:172.21.15.114:6789/0 conn(0x7f45cc0b4330 legacy=0x7f45cc0b4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:11.005 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.021+0000 7f45cb7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:24:11.005 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.021+0000 7f45cb7fe640 1 monclient: found mon.noname-c 2024-03-29T11:24:11.005 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.021+0000 7f45cb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:11.005 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.021+0000 7f45cb7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.176:6789/0 2024-03-29T11:24:11.005 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.021+0000 7f45cb7fe640 1 -- 172.21.15.114:0/752940550 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45cc06e0c0 con 0x7f45cc1470b0 2024-03-29T11:24:11.005 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.021+0000 7f45cb7fe640 10 monclient: _finish_auth 0 2024-03-29T11:24:11.005 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.021+0000 7f45cb7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:24:11.005 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.021+0000 7f45cb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:11.005 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.021+0000 7f45d2b3d640 5 monclient: authenticate success, global_id 4342 2024-03-29T11:24:11.005 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.021+0000 7f45d2b3d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:11.005 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.022+0000 7f45cb7fe640 1 -- 172.21.15.114:0/752940550 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f45c0001670 con 0x7f45cc1470b0 2024-03-29T11:24:11.006 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.022+0000 7f45cb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:11.006 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.022+0000 7f45cb7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:24:11.006 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.022+0000 7f45cb7fe640 10 monclient: dump: 2024-03-29T11:24:11.006 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:24:11.006 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:24:11.006 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:24:11.006 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:24:11.006 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:11.006 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:24:11.007 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:24:11.007 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:24:11.007 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:24:11.007 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:24:11.007 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.022+0000 7f45d2b3d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:11.011 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.022+0000 7f45cb7fe640 1 -- 172.21.15.114:0/752940550 <== mon.1 v1:172.21.15.176:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f45c0001850 con 0x7f45cc1470b0 2024-03-29T11:24:11.011 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.022+0000 7f45cb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:11.012 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.022+0000 7f45d2b3d640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:11.012 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.022+0000 7f45d2b3d640 4 set_mon_vals no callback set 2024-03-29T11:24:11.012 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.022+0000 7f45cb7fe640 1 -- 172.21.15.114:0/752940550 <== mon.1 v1:172.21.15.176:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f45c0001670 con 0x7f45cc1470b0 2024-03-29T11:24:11.012 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.022+0000 7f45cb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:11.012 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.022+0000 7f45cb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:24:11.012 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.022+0000 7f45cb7fe640 10 monclient: dump: 2024-03-29T11:24:11.012 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:24:11.012 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:24:11.012 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:24:11.013 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:24:11.013 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:11.013 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:24:11.013 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:24:11.013 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:24:11.013 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:24:11.013 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:24:11.013 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.023+0000 7f45d2b3d640 10 monclient: shutdown 2024-03-29T11:24:11.013 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.023+0000 7f45d2b3d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:11.014 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.023+0000 7f45d2b3d640 1 -- 172.21.15.114:0/752940550 >> v1:172.21.15.176:6789/0 conn(0x7f45cc1470b0 legacy=0x7f45cc0b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:11.014 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.023+0000 7f45d2b3d640 1 -- 172.21.15.114:0/752940550 shutdown_connections 2024-03-29T11:24:11.014 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.023+0000 7f45d2b3d640 1 -- 172.21.15.114:0/752940550 >> 172.21.15.114:0/752940550 conn(0x7f45cc066a00 msgr2=0x7f45cc066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:11.014 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.023+0000 7f45d2b3d640 1 -- 172.21.15.114:0/752940550 shutdown_connections 2024-03-29T11:24:11.014 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.023+0000 7f45d2b3d640 1 -- 172.21.15.114:0/752940550 wait complete. 2024-03-29T11:24:11.014 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.024+0000 7f45d2b3d640 10 monclient: build_initial_monmap 2024-03-29T11:24:11.014 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.024+0000 7f45d2b3d640 10 monclient: monmap: 2024-03-29T11:24:11.015 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:24:11.015 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:11.015 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T11:24:11.015 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T11:24:11.015 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:11.015 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:24:11.015 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:24:11.015 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T11:24:11.015 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T11:24:11.015 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:24:11.016 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.025+0000 7f45d2b3d640 1 librados: starting msgr at 2024-03-29T11:24:11.016 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.025+0000 7f45d2b3d640 1 librados: starting objecter 2024-03-29T11:24:11.016 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.025+0000 7f45d2b3d640 1 Processor -- start 2024-03-29T11:24:11.016 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.025+0000 7f45d2b3d640 1 -- start start 2024-03-29T11:24:11.016 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.025+0000 7f45d2b3d640 1 librados: setting wanted keys 2024-03-29T11:24:11.016 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.025+0000 7f45d2b3d640 1 librados: calling monclient init 2024-03-29T11:24:11.016 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.025+0000 7f45d2b3d640 10 monclient: init 2024-03-29T11:24:11.016 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.026+0000 7f45d2b3d640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:11.016 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.026+0000 7f45d2b3d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:24:11.017 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.026+0000 7f45d2b3d640 10 monclient(hunting): picked mon.noname-b con 0x7f45cc0b4330 addr v1:172.21.15.176:6789/0 2024-03-29T11:24:11.017 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.026+0000 7f45d2b3d640 10 monclient(hunting): picked mon.noname-a con 0x7f45cc1439b0 addr v1:172.21.15.114:6789/0 2024-03-29T11:24:11.017 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.026+0000 7f45d2b3d640 10 monclient(hunting): picked mon.noname-c con 0x7f45cc1dc3e0 addr v1:172.21.15.114:6790/0 2024-03-29T11:24:11.017 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.026+0000 7f45d2b3d640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45cc1db840 con 0x7f45cc1439b0 2024-03-29T11:24:11.017 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.026+0000 7f45d2b3d640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45cc127460 con 0x7f45cc0b4330 2024-03-29T11:24:11.017 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.026+0000 7f45d2b3d640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45cc122800 con 0x7f45cc1dc3e0 2024-03-29T11:24:11.017 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.026+0000 7f45d2b3d640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:11.017 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.026+0000 7f45d2b3d640 10 monclient(hunting): authenticate will time out at 1121.208846s 2024-03-29T11:24:11.017 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.026+0000 7f45d08d9640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f45cc0b4330 0x7f45cc1dbcd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.114:47644/0 (socket says 172.21.15.114:47644) 2024-03-29T11:24:11.017 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.026+0000 7f45d08d9640 1 -- 172.21.15.114:0/63841713 learned_addr learned my addr 172.21.15.114:0/63841713 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:24:11.017 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.027+0000 7f45c97fa640 10 client.?.objecter ms_handle_connect 0x7f45cc0b4330 2024-03-29T11:24:11.017 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.027+0000 7f45c97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:11.017 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.027+0000 7f45c97fa640 10 client.?.objecter ms_handle_connect 0x7f45cc1dc3e0 2024-03-29T11:24:11.017 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.027+0000 7f45c97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:11.018 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.027+0000 7f45c97fa640 1 -- 172.21.15.114:0/63841713 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2926682741 0 0) 0x7f45cc127460 con 0x7f45cc0b4330 2024-03-29T11:24:11.018 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.027+0000 7f45c97fa640 10 monclient(hunting): _negotiate 2024-03-29T11:24:11.018 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.027+0000 7f45c97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:11.018 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.027+0000 7f45c97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:11.018 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.027+0000 7f45c97fa640 10 monclient(hunting): my global_id is 4345 2024-03-29T11:24:11.018 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.027+0000 7f45c97fa640 1 -- 172.21.15.114:0/63841713 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45b8004eb0 con 0x7f45cc0b4330 2024-03-29T11:24:11.018 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.027+0000 7f45c97fa640 10 client.?.objecter ms_handle_connect 0x7f45cc1439b0 2024-03-29T11:24:11.018 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.027+0000 7f45c97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:11.018 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.027+0000 7f45c97fa640 1 -- 172.21.15.114:0/63841713 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3866516445 0 0) 0x7f45cc122800 con 0x7f45cc1dc3e0 2024-03-29T11:24:11.018 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.027+0000 7f45c97fa640 10 monclient(hunting): _negotiate 2024-03-29T11:24:11.018 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.027+0000 7f45c97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:11.018 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.027+0000 7f45c97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:11.018 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.027+0000 7f45c97fa640 10 monclient(hunting): my global_id is 4352 2024-03-29T11:24:11.018 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.027+0000 7f45c97fa640 1 -- 172.21.15.114:0/63841713 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45cc127460 con 0x7f45cc1dc3e0 2024-03-29T11:24:11.018 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.027+0000 7f45c97fa640 1 -- 172.21.15.114:0/63841713 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 371589073 0 0) 0x7f45b8004eb0 con 0x7f45cc0b4330 2024-03-29T11:24:11.019 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.028+0000 7f45c97fa640 1 -- 172.21.15.114:0/63841713 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f45cc122800 con 0x7f45cc0b4330 2024-03-29T11:24:11.019 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.028+0000 7f45c97fa640 1 -- 172.21.15.114:0/63841713 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f45c4001cb0 con 0x7f45cc0b4330 2024-03-29T11:24:11.019 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.028+0000 7f45c97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:11.019 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.028+0000 7f45c97fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:24:11.019 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.028+0000 7f45c97fa640 10 monclient(hunting): dump: 2024-03-29T11:24:11.019 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:24:11.019 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:24:11.019 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:24:11.019 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:24:11.019 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:11.019 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:24:11.019 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:24:11.019 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:24:11.019 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:24:11.019 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:24:11.020 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.028+0000 7f45c97fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:11.020 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.028+0000 7f45c97fa640 1 -- 172.21.15.114:0/63841713 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2326517443 0 0) 0x7f45cc1db840 con 0x7f45cc1439b0 2024-03-29T11:24:11.020 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.028+0000 7f45c97fa640 10 monclient(hunting): _negotiate 2024-03-29T11:24:11.020 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.028+0000 7f45c97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:11.020 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.028+0000 7f45c97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:11.020 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.028+0000 7f45c97fa640 10 monclient(hunting): my global_id is 4347 2024-03-29T11:24:11.020 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.028+0000 7f45c97fa640 1 -- 172.21.15.114:0/63841713 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45b8004eb0 con 0x7f45cc1439b0 2024-03-29T11:24:11.020 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.028+0000 7f45c97fa640 1 -- 172.21.15.114:0/63841713 <== mon.1 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3582200479 0 0) 0x7f45cc122800 con 0x7f45cc0b4330 2024-03-29T11:24:11.020 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.028+0000 7f45c97fa640 1 -- 172.21.15.114:0/63841713 >> v1:172.21.15.114:6790/0 conn(0x7f45cc1dc3e0 legacy=0x7f45cc125aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:11.020 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.028+0000 7f45c97fa640 1 -- 172.21.15.114:0/63841713 >> v1:172.21.15.114:6789/0 conn(0x7f45cc1439b0 legacy=0x7f45cc141020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:11.020 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.028+0000 7f45c97fa640 10 monclient: _finish_hunting 0 2024-03-29T11:24:11.020 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.028+0000 7f45c97fa640 1 monclient: found mon.b 2024-03-29T11:24:11.020 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.028+0000 7f45c97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:11.021 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.028+0000 7f45c97fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:24:11.021 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.028+0000 7f45c97fa640 1 -- 172.21.15.114:0/63841713 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45cc122990 con 0x7f45cc0b4330 2024-03-29T11:24:11.021 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.028+0000 7f45c97fa640 10 monclient: _finish_auth 0 2024-03-29T11:24:11.021 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.028+0000 7f45c97fa640 10 monclient: _check_auth_tickets 2024-03-29T11:24:11.021 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.028+0000 7f45c97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:11.021 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.028+0000 7f45d2b3d640 5 monclient: authenticate success, global_id 4345 2024-03-29T11:24:11.021 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.028+0000 7f45d2b3d640 10 monclient: _renew_subs 2024-03-29T11:24:11.021 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.028+0000 7f45d2b3d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:24:11.021 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.028+0000 7f45d2b3d640 1 -- 172.21.15.114:0/63841713 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45cc141730 con 0x7f45cc0b4330 2024-03-29T11:24:11.021 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.028+0000 7f45d2b3d640 10 client.4345.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:11.021 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.028+0000 7f45d2b3d640 10 monclient: _renew_subs 2024-03-29T11:24:11.021 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.028+0000 7f45d2b3d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:24:11.021 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.028+0000 7f45d2b3d640 1 -- 172.21.15.114:0/63841713 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f45cc1261b0 con 0x7f45cc0b4330 2024-03-29T11:24:11.021 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.028+0000 7f45d2b3d640 1 librados: init done 2024-03-29T11:24:11.021 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.029+0000 7f45c97fa640 1 -- 172.21.15.114:0/63841713 <== mon.1 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f45c4003990 con 0x7f45cc0b4330 2024-03-29T11:24:11.022 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.029+0000 7f45c97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:11.022 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.029+0000 7f45c97fa640 1 -- 172.21.15.114:0/63841713 <== mon.1 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f45c4003d20 con 0x7f45cc0b4330 2024-03-29T11:24:11.022 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.029+0000 7f45c97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:11.022 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.029+0000 7f45c97fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:24:11.022 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.029+0000 7f45c97fa640 10 monclient: dump: 2024-03-29T11:24:11.022 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:24:11.022 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:24:11.022 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:24:11.022 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:24:11.022 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:11.022 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:24:11.022 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:24:11.022 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:24:11.022 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:24:11.023 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:24:11.023 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.029+0000 7f45caffd640 4 set_mon_vals no callback set 2024-03-29T11:24:11.023 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.030+0000 7f45d2b3d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:11.023 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.030+0000 7f45d2b3d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:11.023 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.030+0000 7f45d2b3d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:24:11.023 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.030+0000 7f45d2b3d640 1 -- 172.21.15.114:0/63841713 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45ac005080 con 0x7f45cc0b4330 2024-03-29T11:24:11.024 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.031+0000 7f45c97fa640 1 -- 172.21.15.114:0/63841713 <== mon.1 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1594292461 0 0) 0x7f45c4003e80 con 0x7f45cc0b4330 2024-03-29T11:24:11.024 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.032+0000 7f45c97fa640 1 -- 172.21.15.114:0/63841713 <== mon.1 v1:172.21.15.176:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3238987313 0 0) 0x7f45c4084d60 con 0x7f45cc0b4330 2024-03-29T11:24:11.024 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.032+0000 7f45c97fa640 10 client.4345.objecter ms_dispatch 0x7f45cc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:24:11.024 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.032+0000 7f45c97fa640 3 client.4345.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:24:11.024 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.032+0000 7f45c97fa640 3 client.4345.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:24:11.024 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.032+0000 7f45cbfff640 10 monclient: get_auth_request con 0x7f45b806de00 auth_method 0 2024-03-29T11:24:11.024 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.032+0000 7f45c97fa640 20 client.4345.objecter dump_active .. 0 homeless 2024-03-29T11:24:11.024 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.033+0000 7f45c97fa640 10 client.4345.objecter ms_handle_connect 0x7f45b806de00 2024-03-29T11:24:11.025 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.039+0000 7f45c97fa640 1 -- 172.21.15.114:0/63841713 <== mon.1 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f45c4050640 con 0x7f45cc0b4330 2024-03-29T11:24:11.025 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.039+0000 7f45c97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:11.025 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.039+0000 7f45c97fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:11.412 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.431+0000 7f45d2b3d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:11.413 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.431+0000 7f45d2b3d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:11.413 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.431+0000 7f45d2b3d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:24:11.413 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.431+0000 7f45d2b3d640 1 -- 172.21.15.114:0/63841713 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f45ac005330 con 0x7f45cc0b4330 2024-03-29T11:24:11.414 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.432+0000 7f45c97fa640 1 -- 172.21.15.114:0/63841713 <== mon.1 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2858224613) 0x7f45c4052570 con 0x7f45cc0b4330 2024-03-29T11:24:11.414 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.432+0000 7f45c97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:11.414 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.432+0000 7f45c97fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:11.414 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T11:24:11.414 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"b0de2b9b-c45a-44c7-b6ee-e83b39a1e108","created":"2024-03-29T11:20:49.420660+0000","modified":"2024-03-29T11:20:51.700167+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:51.700167+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a64785d3-6b11-499a-b706-cc827ab96a74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5882dadb-e7a8-4604-ba9a-418013c106c8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d7e8c409-bc66-4e4a-bbdb-420bad31388e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0682989-3595-4cd6-90a5-ab9111c278e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"67068120-e3ff-4817-bd06-44feb7479a35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09e8b-8a85-4997-bdea-75d754e6ee67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2f0d9ead-f47f-4cb3-92a4-0a5ad23cd831","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fdb6a25f-6bd8-4c65-9b9b-b460ba7673a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:24:11.418 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.436+0000 7f45d2b3d640 10 librados: watch_flush enter 2024-03-29T11:24:11.418 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.437+0000 7f45d2b3d640 10 librados: watch_flush exit 2024-03-29T11:24:11.418 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.437+0000 7f45d2b3d640 20 client.4345.objecter shutdown clearing up homeless session... 2024-03-29T11:24:11.418 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.437+0000 7f45d2b3d640 10 client.4345.objecter successfully canceled tick 2024-03-29T11:24:11.419 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.437+0000 7f45d2b3d640 1 -- 172.21.15.114:0/63841713 >> v1:172.21.15.114:6816/4038755055 conn(0x7f45b806de00 legacy=0x7f45b80702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:11.419 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.437+0000 7f45d2b3d640 10 monclient: shutdown 2024-03-29T11:24:11.419 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.437+0000 7f45d2b3d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:11.419 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.437+0000 7f45d2b3d640 1 -- 172.21.15.114:0/63841713 >> v1:172.21.15.176:6789/0 conn(0x7f45cc0b4330 legacy=0x7f45cc1dbcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:11.419 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.437+0000 7f45d2b3d640 1 -- 172.21.15.114:0/63841713 shutdown_connections 2024-03-29T11:24:11.419 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.437+0000 7f45d2b3d640 1 -- 172.21.15.114:0/63841713 >> 172.21.15.114:0/63841713 conn(0x7f45cc066a00 msgr2=0x7f45cc145ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:11.419 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.437+0000 7f45d2b3d640 1 -- 172.21.15.114:0/63841713 shutdown_connections 2024-03-29T11:24:11.419 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.437+0000 7f45d2b3d640 1 -- 172.21.15.114:0/63841713 wait complete. 2024-03-29T11:24:11.420 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:11.437+0000 7f45d2b3d640 1 librados: shutdown 2024-03-29T11:24:11.435 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:24:18.242 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:24:18.394 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.413+0000 7f65b2843640 10 monclient: get_monmap_and_config 2024-03-29T11:24:18.395 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.413+0000 7f65b2843640 10 monclient: build_initial_monmap 2024-03-29T11:24:18.395 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.413+0000 7f65b2843640 10 monclient: monmap: 2024-03-29T11:24:18.395 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:24:18.395 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:24:18.395 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:24:18.414688+0000 2024-03-29T11:24:18.395 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:24:18.414688+0000 2024-03-29T11:24:18.395 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:18.395 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:24:18.395 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:24:18.395 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T11:24:18.395 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T11:24:18.395 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:24:18.396 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.414+0000 7f65b2843640 1 Processor -- start 2024-03-29T11:24:18.396 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.414+0000 7f65b2843640 1 -- start start 2024-03-29T11:24:18.396 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.414+0000 7f65b2843640 10 monclient: init 2024-03-29T11:24:18.397 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.415+0000 7f65b2843640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:18.397 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.415+0000 7f65b2843640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:24:18.397 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.415+0000 7f65b2843640 10 monclient(hunting): picked mon.noname-a con 0x7f65ac0b1f60 addr v1:172.21.15.114:6789/0 2024-03-29T11:24:18.397 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.415+0000 7f65b2843640 10 monclient(hunting): picked mon.noname-c con 0x7f65ac071eb0 addr v1:172.21.15.176:6789/0 2024-03-29T11:24:18.397 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.415+0000 7f65b2843640 10 monclient(hunting): picked mon.noname-b con 0x7f65ac143d10 addr v1:172.21.15.114:6790/0 2024-03-29T11:24:18.397 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.415+0000 7f65b2843640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65ac06ef80 con 0x7f65ac0b1f60 2024-03-29T11:24:18.397 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.415+0000 7f65b2843640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65ac06f260 con 0x7f65ac071eb0 2024-03-29T11:24:18.397 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.415+0000 7f65b2843640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65ac0b40c0 con 0x7f65ac143d10 2024-03-29T11:24:18.397 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.415+0000 7f65b2843640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:18.397 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.415+0000 7f65b2843640 10 monclient(hunting): authenticate will time out at 1128.597902s 2024-03-29T11:24:18.397 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.416+0000 7f65a3fff640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f65ac071eb0 0x7f65ac146470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.114:45210/0 (socket says 172.21.15.114:45210) 2024-03-29T11:24:18.398 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.416+0000 7f65a3fff640 1 -- 172.21.15.114:0/543340013 learned_addr learned my addr 172.21.15.114:0/543340013 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:24:18.399 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.417+0000 7f65ab7fe640 1 -- 172.21.15.114:0/543340013 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2963550734 0 0) 0x7f65ac0b40c0 con 0x7f65ac143d10 2024-03-29T11:24:18.399 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.417+0000 7f65ab7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:24:18.400 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.417+0000 7f65ab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:18.400 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.417+0000 7f65ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:18.400 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.417+0000 7f65ab7fe640 10 monclient(hunting): my global_id is 4355 2024-03-29T11:24:18.400 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.417+0000 7f65ab7fe640 1 -- 172.21.15.114:0/543340013 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6598003520 con 0x7f65ac143d10 2024-03-29T11:24:18.400 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.417+0000 7f65ab7fe640 1 -- 172.21.15.114:0/543340013 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1084089563 0 0) 0x7f65ac06ef80 con 0x7f65ac0b1f60 2024-03-29T11:24:18.400 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.417+0000 7f65ab7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:24:18.400 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.417+0000 7f65ab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:18.400 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.417+0000 7f65ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:18.400 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.417+0000 7f65ab7fe640 10 monclient(hunting): my global_id is 4350 2024-03-29T11:24:18.400 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.417+0000 7f65ab7fe640 1 -- 172.21.15.114:0/543340013 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65ac0b40c0 con 0x7f65ac0b1f60 2024-03-29T11:24:18.400 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.417+0000 7f65ab7fe640 1 -- 172.21.15.114:0/543340013 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3919568259 0 0) 0x7f65ac06f260 con 0x7f65ac071eb0 2024-03-29T11:24:18.401 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.417+0000 7f65ab7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:24:18.401 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.417+0000 7f65ab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:18.401 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.417+0000 7f65ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:18.401 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.417+0000 7f65ab7fe640 10 monclient(hunting): my global_id is 4348 2024-03-29T11:24:18.401 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.417+0000 7f65ab7fe640 1 -- 172.21.15.114:0/543340013 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65ac06ef80 con 0x7f65ac071eb0 2024-03-29T11:24:18.401 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.417+0000 7f65ab7fe640 1 -- 172.21.15.114:0/543340013 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 566416419 0 0) 0x7f65ac0b40c0 con 0x7f65ac0b1f60 2024-03-29T11:24:18.401 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.417+0000 7f65ab7fe640 1 -- 172.21.15.114:0/543340013 >> v1:172.21.15.114:6790/0 conn(0x7f65ac143d10 legacy=0x7f65ac146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:18.401 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.417+0000 7f65ab7fe640 1 -- 172.21.15.114:0/543340013 >> v1:172.21.15.176:6789/0 conn(0x7f65ac071eb0 legacy=0x7f65ac146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:18.401 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.417+0000 7f65ab7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:24:18.401 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.417+0000 7f65ab7fe640 1 monclient: found mon.noname-a 2024-03-29T11:24:18.401 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.417+0000 7f65ab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:18.401 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.417+0000 7f65ab7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.114:6789/0 2024-03-29T11:24:18.401 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.417+0000 7f65ab7fe640 1 -- 172.21.15.114:0/543340013 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65ac155d70 con 0x7f65ac0b1f60 2024-03-29T11:24:18.402 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.418+0000 7f65ab7fe640 10 monclient: _finish_auth 0 2024-03-29T11:24:18.402 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.418+0000 7f65ab7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:24:18.402 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.418+0000 7f65ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:18.402 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.418+0000 7f65ab7fe640 1 -- 172.21.15.114:0/543340013 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f659c001670 con 0x7f65ac0b1f60 2024-03-29T11:24:18.402 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.418+0000 7f65ab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:18.402 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.418+0000 7f65ab7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:24:18.402 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.418+0000 7f65ab7fe640 10 monclient: dump: 2024-03-29T11:24:18.402 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:24:18.402 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:24:18.402 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:24:18.402 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:24:18.402 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:18.403 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:24:18.403 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:24:18.403 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:24:18.403 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:24:18.403 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:24:18.403 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.418+0000 7f65b2843640 5 monclient: authenticate success, global_id 4350 2024-03-29T11:24:18.403 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.418+0000 7f65b2843640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:18.403 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.418+0000 7f65ab7fe640 1 -- 172.21.15.114:0/543340013 <== mon.0 v1:172.21.15.114:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f659c001890 con 0x7f65ac0b1f60 2024-03-29T11:24:18.403 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.418+0000 7f65ab7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:18.403 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.418+0000 7f65ab7fe640 1 -- 172.21.15.114:0/543340013 <== mon.0 v1:172.21.15.114:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f659c001c20 con 0x7f65ac0b1f60 2024-03-29T11:24:18.403 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.418+0000 7f65b2843640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:18.403 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.418+0000 7f65b2843640 4 set_mon_vals no callback set 2024-03-29T11:24:18.404 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.418+0000 7f65ab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:18.404 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.418+0000 7f65ab7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:24:18.404 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.418+0000 7f65ab7fe640 10 monclient: dump: 2024-03-29T11:24:18.404 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:24:18.404 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:24:18.404 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:24:18.404 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:24:18.404 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:18.404 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:24:18.404 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:24:18.404 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:24:18.404 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:24:18.404 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:24:18.410 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.419+0000 7f65b2843640 10 monclient: shutdown 2024-03-29T11:24:18.410 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.419+0000 7f65b2843640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:18.410 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.419+0000 7f65b2843640 1 -- 172.21.15.114:0/543340013 >> v1:172.21.15.114:6789/0 conn(0x7f65ac0b1f60 legacy=0x7f65ac143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:18.410 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.419+0000 7f65b2843640 1 -- 172.21.15.114:0/543340013 shutdown_connections 2024-03-29T11:24:18.411 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.419+0000 7f65b2843640 1 -- 172.21.15.114:0/543340013 >> 172.21.15.114:0/543340013 conn(0x7f65ac066a00 msgr2=0x7f65ac066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:18.411 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.419+0000 7f65b2843640 1 -- 172.21.15.114:0/543340013 shutdown_connections 2024-03-29T11:24:18.411 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.419+0000 7f65b2843640 1 -- 172.21.15.114:0/543340013 wait complete. 2024-03-29T11:24:18.411 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.420+0000 7f65b2843640 10 monclient: build_initial_monmap 2024-03-29T11:24:18.411 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.420+0000 7f65b2843640 10 monclient: monmap: 2024-03-29T11:24:18.411 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:24:18.411 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:18.411 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T11:24:18.411 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T11:24:18.411 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:18.411 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:24:18.411 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:24:18.412 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T11:24:18.412 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T11:24:18.412 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:24:18.412 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.420+0000 7f65b2843640 1 librados: starting msgr at 2024-03-29T11:24:18.412 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.420+0000 7f65b2843640 1 librados: starting objecter 2024-03-29T11:24:18.412 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.421+0000 7f65b2843640 1 Processor -- start 2024-03-29T11:24:18.412 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.421+0000 7f65b2843640 1 -- start start 2024-03-29T11:24:18.412 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.421+0000 7f65b2843640 1 librados: setting wanted keys 2024-03-29T11:24:18.412 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.421+0000 7f65b2843640 1 librados: calling monclient init 2024-03-29T11:24:18.412 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.421+0000 7f65b2843640 10 monclient: init 2024-03-29T11:24:18.412 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.421+0000 7f65b2843640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:18.412 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.421+0000 7f65b2843640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:24:18.413 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.421+0000 7f65b2843640 10 monclient(hunting): picked mon.noname-b con 0x7f65ac0b1f60 addr v1:172.21.15.176:6789/0 2024-03-29T11:24:18.413 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.421+0000 7f65b2843640 10 monclient(hunting): picked mon.noname-a con 0x7f65ac143d10 addr v1:172.21.15.114:6789/0 2024-03-29T11:24:18.413 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.422+0000 7f65b2843640 10 monclient(hunting): picked mon.noname-c con 0x7f65ac1494b0 addr v1:172.21.15.114:6790/0 2024-03-29T11:24:18.413 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.422+0000 7f65b2843640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65ac14b130 con 0x7f65ac143d10 2024-03-29T11:24:18.413 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.422+0000 7f65b2843640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65ac1dfa70 con 0x7f65ac0b1f60 2024-03-29T11:24:18.413 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.422+0000 7f65b2843640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65ac14b3a0 con 0x7f65ac1494b0 2024-03-29T11:24:18.413 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.422+0000 7f65b2843640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:18.413 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.422+0000 7f65b2843640 10 monclient(hunting): authenticate will time out at 1128.604240s 2024-03-29T11:24:18.413 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.422+0000 7f65b0de0640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f65ac1494b0 0x7f65ac1df360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:52598/0 (socket says 172.21.15.114:52598) 2024-03-29T11:24:18.413 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.422+0000 7f65b0de0640 1 -- 172.21.15.114:0/3672183199 learned_addr learned my addr 172.21.15.114:0/3672183199 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:24:18.413 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.422+0000 7f65a97fa640 10 client.?.objecter ms_handle_connect 0x7f65ac143d10 2024-03-29T11:24:18.414 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.422+0000 7f65a97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:18.414 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.422+0000 7f65a97fa640 10 client.?.objecter ms_handle_connect 0x7f65ac1494b0 2024-03-29T11:24:18.414 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.422+0000 7f65a97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:18.414 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.422+0000 7f65a97fa640 1 -- 172.21.15.114:0/3672183199 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2670451623 0 0) 0x7f65ac14b3a0 con 0x7f65ac1494b0 2024-03-29T11:24:18.414 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.422+0000 7f65a97fa640 10 monclient(hunting): _negotiate 2024-03-29T11:24:18.414 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.422+0000 7f65a97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:18.414 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.422+0000 7f65a97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:18.414 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.422+0000 7f65a97fa640 10 monclient(hunting): my global_id is 4358 2024-03-29T11:24:18.414 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.422+0000 7f65a97fa640 1 -- 172.21.15.114:0/3672183199 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f658c004eb0 con 0x7f65ac1494b0 2024-03-29T11:24:18.414 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.423+0000 7f65a97fa640 10 client.?.objecter ms_handle_connect 0x7f65ac0b1f60 2024-03-29T11:24:18.414 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.423+0000 7f65a97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:18.414 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.423+0000 7f65a97fa640 1 -- 172.21.15.114:0/3672183199 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4055246874 0 0) 0x7f65ac14b130 con 0x7f65ac143d10 2024-03-29T11:24:18.415 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.423+0000 7f65a97fa640 10 monclient(hunting): _negotiate 2024-03-29T11:24:18.415 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.423+0000 7f65a97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:18.415 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.423+0000 7f65a97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:18.415 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.423+0000 7f65a97fa640 10 monclient(hunting): my global_id is 4353 2024-03-29T11:24:18.415 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.423+0000 7f65a97fa640 1 -- 172.21.15.114:0/3672183199 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65ac14b3a0 con 0x7f65ac143d10 2024-03-29T11:24:18.415 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.423+0000 7f65a97fa640 1 -- 172.21.15.114:0/3672183199 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1412680437 0 0) 0x7f658c004eb0 con 0x7f65ac1494b0 2024-03-29T11:24:18.415 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.423+0000 7f65a97fa640 1 -- 172.21.15.114:0/3672183199 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f658c005490 con 0x7f65ac1494b0 2024-03-29T11:24:18.415 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.423+0000 7f65a97fa640 1 -- 172.21.15.114:0/3672183199 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f65a4001c30 con 0x7f65ac1494b0 2024-03-29T11:24:18.415 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.423+0000 7f65a97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:18.415 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.423+0000 7f65a97fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:24:18.415 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.423+0000 7f65a97fa640 10 monclient(hunting): dump: 2024-03-29T11:24:18.415 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:24:18.415 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:24:18.415 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:24:18.416 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:24:18.416 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:18.416 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:24:18.416 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:24:18.416 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:24:18.416 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:24:18.416 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:24:18.416 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.423+0000 7f65a97fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:18.416 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.423+0000 7f65a97fa640 1 -- 172.21.15.114:0/3672183199 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2279128723 0 0) 0x7f65ac14b3a0 con 0x7f65ac143d10 2024-03-29T11:24:18.416 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.423+0000 7f65a97fa640 1 -- 172.21.15.114:0/3672183199 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f658c004eb0 con 0x7f65ac143d10 2024-03-29T11:24:18.416 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.423+0000 7f65a97fa640 1 -- 172.21.15.114:0/3672183199 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f6594001870 con 0x7f65ac143d10 2024-03-29T11:24:18.416 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.423+0000 7f65a97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:18.416 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.423+0000 7f65a97fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:24:18.416 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.423+0000 7f65a97fa640 10 monclient(hunting): dump: 2024-03-29T11:24:18.417 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:24:18.417 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:24:18.417 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:24:18.417 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:24:18.417 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:18.417 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:24:18.417 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:24:18.417 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:24:18.417 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:24:18.417 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:24:18.417 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.424+0000 7f65a97fa640 1 -- 172.21.15.114:0/3672183199 <== mon.2 v1:172.21.15.114:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 235444430 0 0) 0x7f658c005490 con 0x7f65ac1494b0 2024-03-29T11:24:18.417 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.424+0000 7f65a97fa640 1 -- 172.21.15.114:0/3672183199 >> v1:172.21.15.176:6789/0 conn(0x7f65ac0b1f60 legacy=0x7f65ac147350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:18.417 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.424+0000 7f65a97fa640 1 -- 172.21.15.114:0/3672183199 >> v1:172.21.15.114:6789/0 conn(0x7f65ac143d10 legacy=0x7f65ac1dec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:18.417 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.424+0000 7f65a97fa640 10 monclient: _finish_hunting 0 2024-03-29T11:24:18.418 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.424+0000 7f65a97fa640 1 monclient: found mon.c 2024-03-29T11:24:18.418 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.424+0000 7f65a97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:18.418 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.424+0000 7f65a97fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:24:18.418 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.424+0000 7f65a97fa640 1 -- 172.21.15.114:0/3672183199 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65ac149a60 con 0x7f65ac1494b0 2024-03-29T11:24:18.418 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.424+0000 7f65a97fa640 10 monclient: _finish_auth 0 2024-03-29T11:24:18.418 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.424+0000 7f65a97fa640 10 monclient: _check_auth_tickets 2024-03-29T11:24:18.418 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.424+0000 7f65a97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:18.418 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.424+0000 7f65b2843640 5 monclient: authenticate success, global_id 4358 2024-03-29T11:24:18.418 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.424+0000 7f65b2843640 10 monclient: _renew_subs 2024-03-29T11:24:18.418 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.424+0000 7f65b2843640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:24:18.418 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.424+0000 7f65b2843640 1 -- 172.21.15.114:0/3672183199 --> v1:172.21.15.114:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65ac147a60 con 0x7f65ac1494b0 2024-03-29T11:24:18.418 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.424+0000 7f65b2843640 10 client.4358.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:18.418 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.424+0000 7f65b2843640 10 monclient: _renew_subs 2024-03-29T11:24:18.419 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.424+0000 7f65b2843640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:24:18.419 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.424+0000 7f65b2843640 1 -- 172.21.15.114:0/3672183199 --> v1:172.21.15.114:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f65ac154c60 con 0x7f65ac1494b0 2024-03-29T11:24:18.419 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.424+0000 7f65b2843640 1 librados: init done 2024-03-29T11:24:18.419 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.424+0000 7f65a97fa640 1 -- 172.21.15.114:0/3672183199 <== mon.2 v1:172.21.15.114:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f65a40020e0 con 0x7f65ac1494b0 2024-03-29T11:24:18.419 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.424+0000 7f65a97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:18.419 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.424+0000 7f65a97fa640 1 -- 172.21.15.114:0/3672183199 <== mon.2 v1:172.21.15.114:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f65a4003b30 con 0x7f65ac1494b0 2024-03-29T11:24:18.419 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.424+0000 7f65a97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:18.419 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.424+0000 7f65aaffd640 4 set_mon_vals no callback set 2024-03-29T11:24:18.419 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.424+0000 7f65a97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:18.419 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.424+0000 7f65a97fa640 10 monclient: dump: 2024-03-29T11:24:18.419 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:24:18.419 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:24:18.419 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:24:18.419 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:24:18.419 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:18.419 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:24:18.419 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:24:18.420 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:24:18.420 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:24:18.420 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:24:18.420 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.426+0000 7f65b2843640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:18.420 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.426+0000 7f65b2843640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:18.420 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.426+0000 7f65b2843640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:24:18.420 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.427+0000 7f65a97fa640 1 -- 172.21.15.114:0/3672183199 <== mon.2 v1:172.21.15.114:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1594292461 0 0) 0x7f65a4003c90 con 0x7f65ac1494b0 2024-03-29T11:24:18.420 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.426+0000 7f65b2843640 1 -- 172.21.15.114:0/3672183199 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6590005080 con 0x7f65ac1494b0 2024-03-29T11:24:18.420 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.428+0000 7f65a97fa640 1 -- 172.21.15.114:0/3672183199 <== mon.2 v1:172.21.15.114:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3238987313 0 0) 0x7f65a4084ca0 con 0x7f65ac1494b0 2024-03-29T11:24:18.420 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.428+0000 7f65a97fa640 10 client.4358.objecter ms_dispatch 0x7f65ac000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:24:18.420 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.428+0000 7f65a97fa640 3 client.4358.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:24:18.420 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.428+0000 7f65a97fa640 3 client.4358.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:24:18.420 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.428+0000 7f65a97fa640 20 client.4358.objecter dump_active .. 0 homeless 2024-03-29T11:24:18.420 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.428+0000 7f65abfff640 10 monclient: get_auth_request con 0x7f658c06e220 auth_method 0 2024-03-29T11:24:18.420 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.429+0000 7f65a97fa640 10 client.4358.objecter ms_handle_connect 0x7f658c06e220 2024-03-29T11:24:18.421 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.435+0000 7f65a97fa640 1 -- 172.21.15.114:0/3672183199 <== mon.2 v1:172.21.15.114:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f65a4050580 con 0x7f65ac1494b0 2024-03-29T11:24:18.421 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.435+0000 7f65a97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:18.421 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.435+0000 7f65a97fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:18.764 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.783+0000 7f65b2843640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:18.764 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.783+0000 7f65b2843640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:18.764 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.783+0000 7f65b2843640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:24:18.764 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.783+0000 7f65b2843640 1 -- 172.21.15.114:0/3672183199 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6590005330 con 0x7f65ac1494b0 2024-03-29T11:24:18.765 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.784+0000 7f65a97fa640 1 -- 172.21.15.114:0/3672183199 <== mon.2 v1:172.21.15.114:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2858224613) 0x7f65a40524b0 con 0x7f65ac1494b0 2024-03-29T11:24:18.765 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.784+0000 7f65a97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:18.765 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.784+0000 7f65a97fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:18.765 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T11:24:18.765 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"b0de2b9b-c45a-44c7-b6ee-e83b39a1e108","created":"2024-03-29T11:20:49.420660+0000","modified":"2024-03-29T11:20:51.700167+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:51.700167+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a64785d3-6b11-499a-b706-cc827ab96a74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5882dadb-e7a8-4604-ba9a-418013c106c8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d7e8c409-bc66-4e4a-bbdb-420bad31388e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0682989-3595-4cd6-90a5-ab9111c278e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"67068120-e3ff-4817-bd06-44feb7479a35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09e8b-8a85-4997-bdea-75d754e6ee67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2f0d9ead-f47f-4cb3-92a4-0a5ad23cd831","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fdb6a25f-6bd8-4c65-9b9b-b460ba7673a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:24:18.769 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.787+0000 7f65b2843640 10 librados: watch_flush enter 2024-03-29T11:24:18.769 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.787+0000 7f65b2843640 10 librados: watch_flush exit 2024-03-29T11:24:18.769 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.787+0000 7f65b2843640 20 client.4358.objecter shutdown clearing up homeless session... 2024-03-29T11:24:18.769 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.787+0000 7f65b2843640 10 client.4358.objecter successfully canceled tick 2024-03-29T11:24:18.769 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.787+0000 7f65b2843640 1 -- 172.21.15.114:0/3672183199 >> v1:172.21.15.114:6816/4038755055 conn(0x7f658c06e220 legacy=0x7f658c0706e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:18.769 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.788+0000 7f65b2843640 10 monclient: shutdown 2024-03-29T11:24:18.769 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.788+0000 7f65b2843640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:18.769 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.788+0000 7f65b2843640 1 -- 172.21.15.114:0/3672183199 >> v1:172.21.15.114:6790/0 conn(0x7f65ac1494b0 legacy=0x7f65ac1df360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:18.769 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.788+0000 7f65b2843640 1 -- 172.21.15.114:0/3672183199 shutdown_connections 2024-03-29T11:24:18.769 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.788+0000 7f65b2843640 1 -- 172.21.15.114:0/3672183199 >> 172.21.15.114:0/3672183199 conn(0x7f65ac066a00 msgr2=0x7f65ac142890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:18.770 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.788+0000 7f65b2843640 1 -- 172.21.15.114:0/3672183199 shutdown_connections 2024-03-29T11:24:18.770 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.788+0000 7f65b2843640 1 -- 172.21.15.114:0/3672183199 wait complete. 2024-03-29T11:24:18.770 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:18.788+0000 7f65b2843640 1 librados: shutdown 2024-03-29T11:24:18.784 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:24:25.591 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:24:25.722 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.740+0000 7f65d7d8f640 10 monclient: get_monmap_and_config 2024-03-29T11:24:25.722 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.740+0000 7f65d7d8f640 10 monclient: build_initial_monmap 2024-03-29T11:24:25.722 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.740+0000 7f65d7d8f640 10 monclient: monmap: 2024-03-29T11:24:25.722 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:24:25.723 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:24:25.723 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:24:25.741656+0000 2024-03-29T11:24:25.723 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:24:25.741656+0000 2024-03-29T11:24:25.723 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:25.723 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:24:25.723 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:24:25.723 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T11:24:25.723 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T11:24:25.723 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:24:25.726 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.741+0000 7f65d7d8f640 1 Processor -- start 2024-03-29T11:24:25.726 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.741+0000 7f65d7d8f640 1 -- start start 2024-03-29T11:24:25.726 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.741+0000 7f65d7d8f640 10 monclient: init 2024-03-29T11:24:25.726 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.742+0000 7f65d7d8f640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:25.726 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.742+0000 7f65d7d8f640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:24:25.726 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.742+0000 7f65d7d8f640 10 monclient(hunting): picked mon.noname-a con 0x7f65d01422b0 addr v1:172.21.15.114:6789/0 2024-03-29T11:24:25.726 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.742+0000 7f65d7d8f640 10 monclient(hunting): picked mon.noname-b con 0x7f65d0142690 addr v1:172.21.15.114:6790/0 2024-03-29T11:24:25.726 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.742+0000 7f65d7d8f640 10 monclient(hunting): picked mon.noname-c con 0x7f65d00b17d0 addr v1:172.21.15.176:6789/0 2024-03-29T11:24:25.727 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.742+0000 7f65d7d8f640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65d0142c80 con 0x7f65d01422b0 2024-03-29T11:24:25.727 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.742+0000 7f65d7d8f640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65d00aae40 con 0x7f65d00b17d0 2024-03-29T11:24:25.727 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.742+0000 7f65d7d8f640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65d013fbd0 con 0x7f65d0142690 2024-03-29T11:24:25.727 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.742+0000 7f65d7d8f640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:25.727 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.742+0000 7f65d7d8f640 10 monclient(hunting): authenticate will time out at 1135.924997s 2024-03-29T11:24:25.727 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.742+0000 7f65d532a640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f65d0142690 0x7f65d00b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:32820/0 (socket says 172.21.15.114:32820) 2024-03-29T11:24:25.727 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.742+0000 7f65d532a640 1 -- 172.21.15.114:0/1306552915 learned_addr learned my addr 172.21.15.114:0/1306552915 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:24:25.727 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.743+0000 7f65d4b29640 1 -- 172.21.15.114:0/1306552915 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3483083551 0 0) 0x7f65d013fbd0 con 0x7f65d0142690 2024-03-29T11:24:25.727 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.743+0000 7f65d4b29640 10 monclient(hunting): _negotiate 2024-03-29T11:24:25.727 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.743+0000 7f65d4b29640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:25.728 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.743+0000 7f65d4b29640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:25.728 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.743+0000 7f65d4b29640 10 monclient(hunting): my global_id is 4361 2024-03-29T11:24:25.728 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.743+0000 7f65d4b29640 1 -- 172.21.15.114:0/1306552915 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65bc003520 con 0x7f65d0142690 2024-03-29T11:24:25.728 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.743+0000 7f65d4b29640 1 -- 172.21.15.114:0/1306552915 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1752964641 0 0) 0x7f65d00aae40 con 0x7f65d00b17d0 2024-03-29T11:24:25.728 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.743+0000 7f65d4b29640 10 monclient(hunting): _negotiate 2024-03-29T11:24:25.728 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.743+0000 7f65d4b29640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:25.728 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.743+0000 7f65d4b29640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:25.728 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.743+0000 7f65d4b29640 10 monclient(hunting): my global_id is 4351 2024-03-29T11:24:25.728 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.743+0000 7f65d4b29640 1 -- 172.21.15.114:0/1306552915 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65d013fbd0 con 0x7f65d00b17d0 2024-03-29T11:24:25.728 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.743+0000 7f65d4b29640 1 -- 172.21.15.114:0/1306552915 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4288046765 0 0) 0x7f65bc003520 con 0x7f65d0142690 2024-03-29T11:24:25.729 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.743+0000 7f65d4b29640 1 -- 172.21.15.114:0/1306552915 >> v1:172.21.15.176:6789/0 conn(0x7f65d00b17d0 legacy=0x7f65d00b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:25.729 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.743+0000 7f65d4b29640 1 -- 172.21.15.114:0/1306552915 >> v1:172.21.15.114:6789/0 conn(0x7f65d01422b0 legacy=0x7f65d013f4c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:24:25.731 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.746+0000 7f65d4b29640 10 monclient: _finish_hunting 0 2024-03-29T11:24:25.731 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.746+0000 7f65d4b29640 1 monclient: found mon.noname-b 2024-03-29T11:24:25.731 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.746+0000 7f65d4b29640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:25.731 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.746+0000 7f65d4b29640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.114:6790/0 2024-03-29T11:24:25.731 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.746+0000 7f65d4b29640 1 -- 172.21.15.114:0/1306552915 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65d0151940 con 0x7f65d0142690 2024-03-29T11:24:25.732 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.746+0000 7f65d4b29640 10 monclient: _finish_auth 0 2024-03-29T11:24:25.732 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.746+0000 7f65d4b29640 10 monclient: _check_auth_tickets 2024-03-29T11:24:25.732 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.746+0000 7f65d4b29640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:25.732 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.746+0000 7f65d4b29640 1 -- 172.21.15.114:0/1306552915 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f65c0001690 con 0x7f65d0142690 2024-03-29T11:24:25.732 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.746+0000 7f65d4b29640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:25.732 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.746+0000 7f65d4b29640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:24:25.732 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.746+0000 7f65d4b29640 10 monclient: dump: 2024-03-29T11:24:25.732 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:24:25.732 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:24:25.732 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:24:25.732 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:24:25.732 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:25.733 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:24:25.733 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:24:25.733 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:24:25.733 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:24:25.733 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:24:25.733 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.746+0000 7f65d7d8f640 5 monclient: authenticate success, global_id 4361 2024-03-29T11:24:25.733 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.746+0000 7f65d7d8f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:25.733 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.746+0000 7f65d4b29640 1 -- 172.21.15.114:0/1306552915 <== mon.2 v1:172.21.15.114:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f65c00017f0 con 0x7f65d0142690 2024-03-29T11:24:25.733 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.746+0000 7f65d4b29640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:25.733 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.746+0000 7f65d4b29640 1 -- 172.21.15.114:0/1306552915 <== mon.2 v1:172.21.15.114:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f65c0001b80 con 0x7f65d0142690 2024-03-29T11:24:25.733 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.746+0000 7f65d4b29640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:25.734 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.746+0000 7f65d4b29640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:25.734 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.746+0000 7f65d4b29640 10 monclient: dump: 2024-03-29T11:24:25.734 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:24:25.734 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:24:25.734 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:24:25.734 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:24:25.734 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:25.734 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:24:25.734 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:24:25.734 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:24:25.734 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:24:25.734 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:24:25.734 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.746+0000 7f65d7d8f640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:25.734 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.746+0000 7f65d7d8f640 4 set_mon_vals no callback set 2024-03-29T11:24:25.734 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.747+0000 7f65d7d8f640 10 monclient: shutdown 2024-03-29T11:24:25.735 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.747+0000 7f65d7d8f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:25.735 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.747+0000 7f65d7d8f640 1 -- 172.21.15.114:0/1306552915 >> v1:172.21.15.114:6790/0 conn(0x7f65d0142690 legacy=0x7f65d00b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:25.735 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.747+0000 7f65d7d8f640 1 -- 172.21.15.114:0/1306552915 shutdown_connections 2024-03-29T11:24:25.735 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.747+0000 7f65d7d8f640 1 -- 172.21.15.114:0/1306552915 >> 172.21.15.114:0/1306552915 conn(0x7f65d0066a00 msgr2=0x7f65d0066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:25.735 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.748+0000 7f65d7d8f640 1 -- 172.21.15.114:0/1306552915 shutdown_connections 2024-03-29T11:24:25.735 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.748+0000 7f65d7d8f640 1 -- 172.21.15.114:0/1306552915 wait complete. 2024-03-29T11:24:25.735 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.749+0000 7f65d7d8f640 10 monclient: build_initial_monmap 2024-03-29T11:24:25.735 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.749+0000 7f65d7d8f640 10 monclient: monmap: 2024-03-29T11:24:25.735 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:24:25.735 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:25.735 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T11:24:25.735 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T11:24:25.735 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:25.735 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:24:25.735 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:24:25.736 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T11:24:25.736 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T11:24:25.736 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:24:25.736 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.749+0000 7f65d7d8f640 1 librados: starting msgr at 2024-03-29T11:24:25.736 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.749+0000 7f65d7d8f640 1 librados: starting objecter 2024-03-29T11:24:25.736 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.749+0000 7f65d7d8f640 1 Processor -- start 2024-03-29T11:24:25.736 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.749+0000 7f65d7d8f640 1 -- start start 2024-03-29T11:24:25.736 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.749+0000 7f65d7d8f640 1 librados: setting wanted keys 2024-03-29T11:24:25.736 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.749+0000 7f65d7d8f640 1 librados: calling monclient init 2024-03-29T11:24:25.736 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.749+0000 7f65d7d8f640 10 monclient: init 2024-03-29T11:24:25.742 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.750+0000 7f65d7d8f640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:25.742 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.750+0000 7f65d7d8f640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:24:25.742 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.750+0000 7f65d7d8f640 10 monclient(hunting): picked mon.noname-a con 0x7f65d00b17d0 addr v1:172.21.15.114:6789/0 2024-03-29T11:24:25.742 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.750+0000 7f65d7d8f640 10 monclient(hunting): picked mon.noname-b con 0x7f65d01422b0 addr v1:172.21.15.176:6789/0 2024-03-29T11:24:25.742 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.750+0000 7f65d632c640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f65d0142690 0x7f65d01db180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:33774/0 (socket says 172.21.15.114:33774) 2024-03-29T11:24:25.742 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.750+0000 7f65d632c640 1 -- 172.21.15.114:0/30489137 learned_addr learned my addr 172.21.15.114:0/30489137 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:24:25.742 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.750+0000 7f65d7d8f640 10 monclient(hunting): picked mon.noname-c con 0x7f65d0142690 addr v1:172.21.15.114:6790/0 2024-03-29T11:24:25.742 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.750+0000 7f65c67fc640 10 client.?.objecter ms_handle_connect 0x7f65d0142690 2024-03-29T11:24:25.742 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.751+0000 7f65c67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:25.742 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.751+0000 7f65d7d8f640 1 -- 172.21.15.114:0/30489137 --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65d00af5f0 con 0x7f65d00b17d0 2024-03-29T11:24:25.743 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.751+0000 7f65d7d8f640 1 -- 172.21.15.114:0/30489137 --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65d00b1060 con 0x7f65d01422b0 2024-03-29T11:24:25.743 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.751+0000 7f65d7d8f640 1 -- 172.21.15.114:0/30489137 --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65d00b0090 con 0x7f65d0142690 2024-03-29T11:24:25.743 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.751+0000 7f65d7d8f640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:25.743 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.751+0000 7f65d7d8f640 10 monclient(hunting): authenticate will time out at 1135.933559s 2024-03-29T11:24:25.743 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.751+0000 7f65c67fc640 1 -- 172.21.15.114:0/30489137 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1387391424 0 0) 0x7f65d00b0090 con 0x7f65d0142690 2024-03-29T11:24:25.743 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.751+0000 7f65c67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:24:25.743 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.751+0000 7f65c67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:25.743 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.751+0000 7f65c67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:25.743 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.751+0000 7f65c67fc640 10 monclient(hunting): my global_id is 4364 2024-03-29T11:24:25.743 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.751+0000 7f65c67fc640 1 -- 172.21.15.114:0/30489137 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65b4004eb0 con 0x7f65d0142690 2024-03-29T11:24:25.743 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.752+0000 7f65c67fc640 10 client.?.objecter ms_handle_connect 0x7f65d01422b0 2024-03-29T11:24:25.743 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.752+0000 7f65c67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:25.743 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.752+0000 7f65c67fc640 1 -- 172.21.15.114:0/30489137 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3854447569 0 0) 0x7f65b4004eb0 con 0x7f65d0142690 2024-03-29T11:24:25.743 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.752+0000 7f65c67fc640 1 -- 172.21.15.114:0/30489137 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f65d00b0090 con 0x7f65d0142690 2024-03-29T11:24:25.744 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.752+0000 7f65c67fc640 1 -- 172.21.15.114:0/30489137 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f65c8001d10 con 0x7f65d0142690 2024-03-29T11:24:25.744 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.752+0000 7f65c67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:25.744 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.752+0000 7f65c67fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:24:25.744 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.752+0000 7f65c67fc640 10 monclient(hunting): dump: 2024-03-29T11:24:25.744 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:24:25.744 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:24:25.744 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:24:25.744 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:24:25.744 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:25.744 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:24:25.744 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:24:25.744 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:24:25.744 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:24:25.744 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:24:25.744 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.752+0000 7f65c67fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:25.745 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.752+0000 7f65c67fc640 1 -- 172.21.15.114:0/30489137 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3470166567 0 0) 0x7f65d00b1060 con 0x7f65d01422b0 2024-03-29T11:24:25.745 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.752+0000 7f65c67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:24:25.745 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.752+0000 7f65c67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:25.745 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.752+0000 7f65c67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:25.745 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.752+0000 7f65c67fc640 10 monclient(hunting): my global_id is 4354 2024-03-29T11:24:25.745 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.752+0000 7f65c67fc640 1 -- 172.21.15.114:0/30489137 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65b4004eb0 con 0x7f65d01422b0 2024-03-29T11:24:25.745 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.752+0000 7f65c67fc640 1 -- 172.21.15.114:0/30489137 <== mon.2 v1:172.21.15.114:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4167825314 0 0) 0x7f65d00b0090 con 0x7f65d0142690 2024-03-29T11:24:25.745 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.752+0000 7f65c67fc640 1 -- 172.21.15.114:0/30489137 >> v1:172.21.15.176:6789/0 conn(0x7f65d01422b0 legacy=0x7f65d01daa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:25.745 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.752+0000 7f65c67fc640 1 -- 172.21.15.114:0/30489137 >> v1:172.21.15.114:6789/0 conn(0x7f65d00b17d0 legacy=0x7f65d00aeee0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:24:25.745 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.753+0000 7f65c67fc640 10 monclient: _finish_hunting 0 2024-03-29T11:24:25.745 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.753+0000 7f65c67fc640 1 monclient: found mon.c 2024-03-29T11:24:25.745 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.753+0000 7f65c67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:25.745 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.753+0000 7f65c67fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:24:25.745 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.753+0000 7f65c67fc640 1 -- 172.21.15.114:0/30489137 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65d00afc60 con 0x7f65d0142690 2024-03-29T11:24:25.745 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.753+0000 7f65c67fc640 10 monclient: _finish_auth 0 2024-03-29T11:24:25.746 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.753+0000 7f65c67fc640 10 monclient: _check_auth_tickets 2024-03-29T11:24:25.746 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.753+0000 7f65c67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:25.746 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.753+0000 7f65d7d8f640 5 monclient: authenticate success, global_id 4364 2024-03-29T11:24:25.746 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.753+0000 7f65d7d8f640 10 monclient: _renew_subs 2024-03-29T11:24:25.746 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.753+0000 7f65d7d8f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:24:25.746 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.753+0000 7f65d7d8f640 1 -- 172.21.15.114:0/30489137 --> v1:172.21.15.114:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65d014d7a0 con 0x7f65d0142690 2024-03-29T11:24:25.746 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.753+0000 7f65d7d8f640 10 client.4364.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:25.746 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.753+0000 7f65d7d8f640 10 monclient: _renew_subs 2024-03-29T11:24:25.746 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.753+0000 7f65d7d8f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:24:25.746 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.753+0000 7f65d7d8f640 1 -- 172.21.15.114:0/30489137 --> v1:172.21.15.114:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f65d014dc80 con 0x7f65d0142690 2024-03-29T11:24:25.746 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.753+0000 7f65d7d8f640 1 librados: init done 2024-03-29T11:24:25.746 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.754+0000 7f65c67fc640 1 -- 172.21.15.114:0/30489137 <== mon.2 v1:172.21.15.114:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f65c8001fd0 con 0x7f65d0142690 2024-03-29T11:24:25.746 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.754+0000 7f65c67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:25.746 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.754+0000 7f65c67fc640 1 -- 172.21.15.114:0/30489137 <== mon.2 v1:172.21.15.114:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f65c8003de0 con 0x7f65d0142690 2024-03-29T11:24:25.747 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.754+0000 7f65c67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:25.747 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.754+0000 7f65c67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:25.747 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.754+0000 7f65c67fc640 10 monclient: dump: 2024-03-29T11:24:25.747 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:24:25.747 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:24:25.747 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:24:25.747 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:24:25.747 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:25.747 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:24:25.747 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:24:25.747 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:24:25.747 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:24:25.747 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:24:25.747 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.754+0000 7f65c77fe640 4 set_mon_vals no callback set 2024-03-29T11:24:25.748 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.755+0000 7f65d7d8f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:25.748 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.755+0000 7f65d7d8f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:25.748 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.755+0000 7f65d7d8f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:24:25.748 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.755+0000 7f65d7d8f640 1 -- 172.21.15.114:0/30489137 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65b8005080 con 0x7f65d0142690 2024-03-29T11:24:25.748 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.755+0000 7f65c67fc640 1 -- 172.21.15.114:0/30489137 <== mon.2 v1:172.21.15.114:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1594292461 0 0) 0x7f65c8003f40 con 0x7f65d0142690 2024-03-29T11:24:25.748 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.756+0000 7f65c67fc640 1 -- 172.21.15.114:0/30489137 <== mon.2 v1:172.21.15.114:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3238987313 0 0) 0x7f65c8084e80 con 0x7f65d0142690 2024-03-29T11:24:25.748 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.756+0000 7f65c67fc640 10 client.4364.objecter ms_dispatch 0x7f65d0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:24:25.748 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.756+0000 7f65c67fc640 3 client.4364.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:24:25.748 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.756+0000 7f65c67fc640 3 client.4364.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:24:25.748 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.756+0000 7f65c67fc640 20 client.4364.objecter dump_active .. 0 homeless 2024-03-29T11:24:25.748 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.756+0000 7f65d5b2b640 10 monclient: get_auth_request con 0x7f65b406dd00 auth_method 0 2024-03-29T11:24:25.748 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.756+0000 7f65c67fc640 10 client.4364.objecter ms_handle_connect 0x7f65b406dd00 2024-03-29T11:24:25.749 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.763+0000 7f65c67fc640 1 -- 172.21.15.114:0/30489137 <== mon.2 v1:172.21.15.114:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f65c8050760 con 0x7f65d0142690 2024-03-29T11:24:25.749 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.763+0000 7f65c67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:25.749 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:25.763+0000 7f65c67fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:26.051 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:26.069+0000 7f65d7d8f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:26.051 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:26.069+0000 7f65d7d8f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:26.051 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:26.069+0000 7f65d7d8f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:24:26.051 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:26.069+0000 7f65d7d8f640 1 -- 172.21.15.114:0/30489137 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f65b8005330 con 0x7f65d0142690 2024-03-29T11:24:26.051 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:26.069+0000 7f65c67fc640 1 -- 172.21.15.114:0/30489137 <== mon.2 v1:172.21.15.114:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2858224613) 0x7f65c8052690 con 0x7f65d0142690 2024-03-29T11:24:26.051 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:26.069+0000 7f65c67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:26.051 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:26.069+0000 7f65c67fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:26.052 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T11:24:26.052 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"b0de2b9b-c45a-44c7-b6ee-e83b39a1e108","created":"2024-03-29T11:20:49.420660+0000","modified":"2024-03-29T11:20:51.700167+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:51.700167+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a64785d3-6b11-499a-b706-cc827ab96a74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5882dadb-e7a8-4604-ba9a-418013c106c8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d7e8c409-bc66-4e4a-bbdb-420bad31388e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0682989-3595-4cd6-90a5-ab9111c278e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"67068120-e3ff-4817-bd06-44feb7479a35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09e8b-8a85-4997-bdea-75d754e6ee67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2f0d9ead-f47f-4cb3-92a4-0a5ad23cd831","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fdb6a25f-6bd8-4c65-9b9b-b460ba7673a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:24:26.054 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:26.072+0000 7f65d7d8f640 10 librados: watch_flush enter 2024-03-29T11:24:26.054 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:26.072+0000 7f65d7d8f640 10 librados: watch_flush exit 2024-03-29T11:24:26.054 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:26.072+0000 7f65d7d8f640 20 client.4364.objecter shutdown clearing up homeless session... 2024-03-29T11:24:26.054 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:26.072+0000 7f65d7d8f640 10 client.4364.objecter successfully canceled tick 2024-03-29T11:24:26.054 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:26.072+0000 7f65d7d8f640 1 -- 172.21.15.114:0/30489137 >> v1:172.21.15.114:6816/4038755055 conn(0x7f65b406dd00 legacy=0x7f65b40701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:26.054 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:26.072+0000 7f65d7d8f640 10 monclient: shutdown 2024-03-29T11:24:26.054 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:26.072+0000 7f65d7d8f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:26.054 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:26.072+0000 7f65d7d8f640 1 -- 172.21.15.114:0/30489137 >> v1:172.21.15.114:6790/0 conn(0x7f65d0142690 legacy=0x7f65d01db180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:26.055 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:26.072+0000 7f65d7d8f640 1 -- 172.21.15.114:0/30489137 shutdown_connections 2024-03-29T11:24:26.055 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:26.072+0000 7f65d7d8f640 1 -- 172.21.15.114:0/30489137 >> 172.21.15.114:0/30489137 conn(0x7f65d0066a00 msgr2=0x7f65d0066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:26.055 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:26.073+0000 7f65d7d8f640 1 -- 172.21.15.114:0/30489137 shutdown_connections 2024-03-29T11:24:26.055 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:26.073+0000 7f65d7d8f640 1 -- 172.21.15.114:0/30489137 wait complete. 2024-03-29T11:24:26.055 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:26.073+0000 7f65d7d8f640 1 librados: shutdown 2024-03-29T11:24:26.069 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:24:32.876 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:24:33.021 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.039+0000 7ff4df331640 10 monclient: get_monmap_and_config 2024-03-29T11:24:33.021 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.039+0000 7ff4df331640 10 monclient: build_initial_monmap 2024-03-29T11:24:33.021 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.039+0000 7ff4df331640 10 monclient: monmap: 2024-03-29T11:24:33.021 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:24:33.021 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:24:33.021 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:24:33.040197+0000 2024-03-29T11:24:33.021 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:24:33.040197+0000 2024-03-29T11:24:33.022 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:33.022 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:24:33.022 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:24:33.022 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T11:24:33.022 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T11:24:33.022 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:24:33.022 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.040+0000 7ff4df331640 1 Processor -- start 2024-03-29T11:24:33.022 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.040+0000 7ff4df331640 1 -- start start 2024-03-29T11:24:33.023 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.040+0000 7ff4df331640 10 monclient: init 2024-03-29T11:24:33.023 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.041+0000 7ff4df331640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:33.023 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.041+0000 7ff4df331640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:24:33.024 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.041+0000 7ff4df331640 10 monclient(hunting): picked mon.noname-c con 0x7ff4d80b26b0 addr v1:172.21.15.176:6789/0 2024-03-29T11:24:33.024 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.042+0000 7ff4ddb2e640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7ff4d80b2a90 0x7ff4d8141e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:53736/0 (socket says 172.21.15.114:53736) 2024-03-29T11:24:33.024 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.042+0000 7ff4ddb2e640 1 -- 172.21.15.114:0/810614925 learned_addr learned my addr 172.21.15.114:0/810614925 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:24:33.024 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.042+0000 7ff4df331640 10 monclient(hunting): picked mon.noname-b con 0x7ff4d80b2a90 addr v1:172.21.15.114:6790/0 2024-03-29T11:24:33.025 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.042+0000 7ff4df331640 10 monclient(hunting): picked mon.noname-a con 0x7ff4d813f5a0 addr v1:172.21.15.114:6789/0 2024-03-29T11:24:33.025 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.042+0000 7ff4df331640 1 -- 172.21.15.114:0/810614925 --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4d80aac60 con 0x7ff4d813f5a0 2024-03-29T11:24:33.025 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.042+0000 7ff4df331640 1 -- 172.21.15.114:0/810614925 --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4d80aaf40 con 0x7ff4d80b26b0 2024-03-29T11:24:33.025 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.042+0000 7ff4df331640 1 -- 172.21.15.114:0/810614925 --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4d80b2f80 con 0x7ff4d80b2a90 2024-03-29T11:24:33.025 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.042+0000 7ff4df331640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:33.025 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.042+0000 7ff4df331640 10 monclient(hunting): authenticate will time out at 1143.224936s 2024-03-29T11:24:33.026 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.044+0000 7ff4dd32d640 1 -- 172.21.15.114:0/810614925 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2106939394 0 0) 0x7ff4d80b2f80 con 0x7ff4d80b2a90 2024-03-29T11:24:33.026 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.044+0000 7ff4dd32d640 10 monclient(hunting): _negotiate 2024-03-29T11:24:33.026 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.044+0000 7ff4dd32d640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:33.026 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.044+0000 7ff4dd32d640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:33.026 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.044+0000 7ff4dd32d640 10 monclient(hunting): my global_id is 4367 2024-03-29T11:24:33.027 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.044+0000 7ff4dd32d640 1 -- 172.21.15.114:0/810614925 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4c4003520 con 0x7ff4d80b2a90 2024-03-29T11:24:33.033 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.045+0000 7ff4dd32d640 1 -- 172.21.15.114:0/810614925 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1690030595 0 0) 0x7ff4d80aaf40 con 0x7ff4d80b26b0 2024-03-29T11:24:33.033 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.045+0000 7ff4dd32d640 10 monclient(hunting): _negotiate 2024-03-29T11:24:33.033 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.045+0000 7ff4dd32d640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:33.034 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.045+0000 7ff4dd32d640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:33.034 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.045+0000 7ff4dd32d640 10 monclient(hunting): my global_id is 4357 2024-03-29T11:24:33.034 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.045+0000 7ff4dd32d640 1 -- 172.21.15.114:0/810614925 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4d80b2f80 con 0x7ff4d80b26b0 2024-03-29T11:24:33.034 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.045+0000 7ff4dd32d640 1 -- 172.21.15.114:0/810614925 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 434045117 0 0) 0x7ff4c4003520 con 0x7ff4d80b2a90 2024-03-29T11:24:33.034 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.045+0000 7ff4dd32d640 1 -- 172.21.15.114:0/810614925 >> v1:172.21.15.176:6789/0 conn(0x7ff4d80b26b0 legacy=0x7ff4d813ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:33.034 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.045+0000 7ff4dd32d640 1 -- 172.21.15.114:0/810614925 >> v1:172.21.15.114:6789/0 conn(0x7ff4d813f5a0 legacy=0x7ff4d8142580 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:24:33.034 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.045+0000 7ff4dd32d640 10 monclient: _finish_hunting 0 2024-03-29T11:24:33.034 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.045+0000 7ff4dd32d640 1 monclient: found mon.noname-b 2024-03-29T11:24:33.034 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.045+0000 7ff4dd32d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:33.035 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.045+0000 7ff4dd32d640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.114:6790/0 2024-03-29T11:24:33.035 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.045+0000 7ff4dd32d640 1 -- 172.21.15.114:0/810614925 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4d80a9be0 con 0x7ff4d80b2a90 2024-03-29T11:24:33.035 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.045+0000 7ff4dd32d640 10 monclient: _finish_auth 0 2024-03-29T11:24:33.035 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.046+0000 7ff4dd32d640 10 monclient: _check_auth_tickets 2024-03-29T11:24:33.035 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.046+0000 7ff4dd32d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:33.035 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.046+0000 7ff4dd32d640 1 -- 172.21.15.114:0/810614925 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7ff4c8001690 con 0x7ff4d80b2a90 2024-03-29T11:24:33.035 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.046+0000 7ff4dd32d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:33.035 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.046+0000 7ff4dd32d640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:24:33.035 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.046+0000 7ff4dd32d640 10 monclient: dump: 2024-03-29T11:24:33.035 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:24:33.035 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:24:33.035 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:24:33.036 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:24:33.036 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:33.036 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:24:33.036 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:24:33.036 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:24:33.036 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:24:33.036 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:24:33.036 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.046+0000 7ff4df331640 5 monclient: authenticate success, global_id 4367 2024-03-29T11:24:33.036 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.046+0000 7ff4df331640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:33.036 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.046+0000 7ff4dd32d640 1 -- 172.21.15.114:0/810614925 <== mon.2 v1:172.21.15.114:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff4c80017f0 con 0x7ff4d80b2a90 2024-03-29T11:24:33.036 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.046+0000 7ff4dd32d640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:33.036 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.046+0000 7ff4dd32d640 1 -- 172.21.15.114:0/810614925 <== mon.2 v1:172.21.15.114:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7ff4c8001c00 con 0x7ff4d80b2a90 2024-03-29T11:24:33.037 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.046+0000 7ff4dd32d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:33.037 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.046+0000 7ff4dd32d640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:33.037 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.046+0000 7ff4dd32d640 10 monclient: dump: 2024-03-29T11:24:33.037 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:24:33.037 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:24:33.037 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:24:33.037 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:24:33.037 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:33.037 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:24:33.037 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:24:33.037 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:24:33.037 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:24:33.037 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:24:33.038 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.046+0000 7ff4df331640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:33.038 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.046+0000 7ff4df331640 4 set_mon_vals no callback set 2024-03-29T11:24:33.038 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.047+0000 7ff4df331640 10 monclient: shutdown 2024-03-29T11:24:33.038 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.047+0000 7ff4df331640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:33.038 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.047+0000 7ff4df331640 1 -- 172.21.15.114:0/810614925 >> v1:172.21.15.114:6790/0 conn(0x7ff4d80b2a90 legacy=0x7ff4d8141e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:33.038 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.048+0000 7ff4df331640 1 -- 172.21.15.114:0/810614925 shutdown_connections 2024-03-29T11:24:33.038 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.048+0000 7ff4df331640 1 -- 172.21.15.114:0/810614925 >> 172.21.15.114:0/810614925 conn(0x7ff4d806d240 msgr2=0x7ff4d8066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:33.038 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.049+0000 7ff4df331640 1 -- 172.21.15.114:0/810614925 shutdown_connections 2024-03-29T11:24:33.038 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.049+0000 7ff4df331640 1 -- 172.21.15.114:0/810614925 wait complete. 2024-03-29T11:24:33.038 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.050+0000 7ff4df331640 10 monclient: build_initial_monmap 2024-03-29T11:24:33.038 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.050+0000 7ff4df331640 10 monclient: monmap: 2024-03-29T11:24:33.038 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:24:33.039 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:33.039 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T11:24:33.039 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T11:24:33.039 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:33.039 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:24:33.039 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:24:33.039 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T11:24:33.039 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T11:24:33.039 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:24:33.039 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.051+0000 7ff4df331640 1 librados: starting msgr at 2024-03-29T11:24:33.039 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.051+0000 7ff4df331640 1 librados: starting objecter 2024-03-29T11:24:33.040 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.051+0000 7ff4df331640 1 Processor -- start 2024-03-29T11:24:33.047 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.052+0000 7ff4df331640 1 -- start start 2024-03-29T11:24:33.047 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.052+0000 7ff4df331640 1 librados: setting wanted keys 2024-03-29T11:24:33.047 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.052+0000 7ff4df331640 1 librados: calling monclient init 2024-03-29T11:24:33.047 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.052+0000 7ff4df331640 10 monclient: init 2024-03-29T11:24:33.047 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.053+0000 7ff4df331640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:33.048 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.053+0000 7ff4df331640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:24:33.048 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.053+0000 7ff4df331640 10 monclient(hunting): picked mon.noname-c con 0x7ff4d80b26b0 addr v1:172.21.15.114:6790/0 2024-03-29T11:24:33.048 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.054+0000 7ff4de32f640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7ff4d80b26b0 0x7ff4d813cd40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:55768/0 (socket says 172.21.15.114:55768) 2024-03-29T11:24:33.048 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.054+0000 7ff4de32f640 1 -- 172.21.15.114:0/625970962 learned_addr learned my addr 172.21.15.114:0/625970962 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:24:33.048 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.054+0000 7ff4df331640 10 monclient(hunting): picked mon.noname-a con 0x7ff4d80b2a90 addr v1:172.21.15.114:6789/0 2024-03-29T11:24:33.048 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.054+0000 7ff4ceffd640 10 client.?.objecter ms_handle_connect 0x7ff4d80b26b0 2024-03-29T11:24:33.048 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.054+0000 7ff4ceffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:33.049 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.057+0000 7ff4df331640 10 monclient(hunting): picked mon.noname-b con 0x7ff4d813f5a0 addr v1:172.21.15.176:6789/0 2024-03-29T11:24:33.049 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.057+0000 7ff4df331640 1 -- 172.21.15.114:0/625970962 --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4d813c110 con 0x7ff4d80b2a90 2024-03-29T11:24:33.049 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.057+0000 7ff4df331640 1 -- 172.21.15.114:0/625970962 --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4d813ace0 con 0x7ff4d813f5a0 2024-03-29T11:24:33.049 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.057+0000 7ff4ceffd640 10 client.?.objecter ms_handle_connect 0x7ff4d813f5a0 2024-03-29T11:24:33.049 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.057+0000 7ff4ceffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:33.049 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.057+0000 7ff4df331640 1 -- 172.21.15.114:0/625970962 --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4d813c360 con 0x7ff4d80b26b0 2024-03-29T11:24:33.049 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.057+0000 7ff4df331640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:33.049 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.057+0000 7ff4df331640 10 monclient(hunting): authenticate will time out at 1143.239823s 2024-03-29T11:24:33.049 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.057+0000 7ff4ceffd640 1 -- 172.21.15.114:0/625970962 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2195093582 0 0) 0x7ff4d813ace0 con 0x7ff4d813f5a0 2024-03-29T11:24:33.050 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.057+0000 7ff4ceffd640 10 monclient(hunting): _negotiate 2024-03-29T11:24:33.050 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.057+0000 7ff4ceffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:33.050 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.057+0000 7ff4ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:33.050 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.057+0000 7ff4ceffd640 10 monclient(hunting): my global_id is 4360 2024-03-29T11:24:33.050 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.057+0000 7ff4ceffd640 1 -- 172.21.15.114:0/625970962 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4bc004eb0 con 0x7ff4d813f5a0 2024-03-29T11:24:33.050 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.058+0000 7ff4ceffd640 10 client.?.objecter ms_handle_connect 0x7ff4d80b2a90 2024-03-29T11:24:33.050 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.058+0000 7ff4ceffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:33.050 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.058+0000 7ff4ceffd640 1 -- 172.21.15.114:0/625970962 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2286582208 0 0) 0x7ff4bc004eb0 con 0x7ff4d813f5a0 2024-03-29T11:24:33.050 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.058+0000 7ff4ceffd640 1 -- 172.21.15.114:0/625970962 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff4d813ace0 con 0x7ff4d813f5a0 2024-03-29T11:24:33.050 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.058+0000 7ff4ceffd640 1 -- 172.21.15.114:0/625970962 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7ff4d0001ad0 con 0x7ff4d813f5a0 2024-03-29T11:24:33.050 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.058+0000 7ff4ceffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:33.050 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.058+0000 7ff4ceffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:24:33.051 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.058+0000 7ff4ceffd640 10 monclient(hunting): dump: 2024-03-29T11:24:33.051 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:24:33.051 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:24:33.051 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:24:33.051 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:24:33.051 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:33.051 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:24:33.051 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:24:33.051 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:24:33.051 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:24:33.051 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:24:33.051 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.058+0000 7ff4ceffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:33.051 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.058+0000 7ff4ceffd640 1 -- 172.21.15.114:0/625970962 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3127966654 0 0) 0x7ff4d813c110 con 0x7ff4d80b2a90 2024-03-29T11:24:33.052 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.058+0000 7ff4ceffd640 10 monclient(hunting): _negotiate 2024-03-29T11:24:33.052 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.058+0000 7ff4ceffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:33.052 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.058+0000 7ff4ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:33.052 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.058+0000 7ff4ceffd640 10 monclient(hunting): my global_id is 4356 2024-03-29T11:24:33.052 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.058+0000 7ff4ceffd640 1 -- 172.21.15.114:0/625970962 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4bc005680 con 0x7ff4d80b2a90 2024-03-29T11:24:33.052 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.059+0000 7ff4ceffd640 1 -- 172.21.15.114:0/625970962 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2728825614 0 0) 0x7ff4d813c360 con 0x7ff4d80b26b0 2024-03-29T11:24:33.052 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.059+0000 7ff4ceffd640 10 monclient(hunting): _negotiate 2024-03-29T11:24:33.052 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.059+0000 7ff4ceffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:33.052 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.059+0000 7ff4ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:33.052 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.059+0000 7ff4ceffd640 10 monclient(hunting): my global_id is 4370 2024-03-29T11:24:33.052 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.059+0000 7ff4ceffd640 1 -- 172.21.15.114:0/625970962 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4d813c110 con 0x7ff4d80b26b0 2024-03-29T11:24:33.052 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.059+0000 7ff4ceffd640 1 -- 172.21.15.114:0/625970962 <== mon.1 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3026322941 0 0) 0x7ff4d813ace0 con 0x7ff4d813f5a0 2024-03-29T11:24:33.052 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.059+0000 7ff4ceffd640 1 -- 172.21.15.114:0/625970962 >> v1:172.21.15.114:6790/0 conn(0x7ff4d80b26b0 legacy=0x7ff4d813cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:33.053 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.059+0000 7ff4ceffd640 1 -- 172.21.15.114:0/625970962 >> v1:172.21.15.114:6789/0 conn(0x7ff4d80b2a90 legacy=0x7ff4d813b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:33.053 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.059+0000 7ff4ceffd640 10 monclient: _finish_hunting 0 2024-03-29T11:24:33.053 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.059+0000 7ff4ceffd640 1 monclient: found mon.b 2024-03-29T11:24:33.053 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.059+0000 7ff4ceffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:33.053 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.059+0000 7ff4ceffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:24:33.053 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.059+0000 7ff4ceffd640 1 -- 172.21.15.114:0/625970962 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4d813c640 con 0x7ff4d813f5a0 2024-03-29T11:24:33.053 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.059+0000 7ff4ceffd640 10 monclient: _finish_auth 0 2024-03-29T11:24:33.053 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.059+0000 7ff4ceffd640 10 monclient: _check_auth_tickets 2024-03-29T11:24:33.053 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.059+0000 7ff4ceffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:33.053 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.059+0000 7ff4df331640 5 monclient: authenticate success, global_id 4360 2024-03-29T11:24:33.053 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.059+0000 7ff4df331640 10 monclient: _renew_subs 2024-03-29T11:24:33.054 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.059+0000 7ff4df331640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:24:33.054 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.059+0000 7ff4df331640 1 -- 172.21.15.114:0/625970962 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4d813c920 con 0x7ff4d813f5a0 2024-03-29T11:24:33.054 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.059+0000 7ff4df331640 10 client.4360.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:33.054 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.059+0000 7ff4df331640 10 monclient: _renew_subs 2024-03-29T11:24:33.054 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.059+0000 7ff4df331640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:24:33.054 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.059+0000 7ff4df331640 1 -- 172.21.15.114:0/625970962 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4d813a4a0 con 0x7ff4d813f5a0 2024-03-29T11:24:33.054 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.059+0000 7ff4df331640 1 librados: init done 2024-03-29T11:24:33.054 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.062+0000 7ff4ceffd640 1 -- 172.21.15.114:0/625970962 <== mon.1 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff4d0002060 con 0x7ff4d813f5a0 2024-03-29T11:24:33.054 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.062+0000 7ff4ceffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:33.054 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.062+0000 7ff4ceffd640 1 -- 172.21.15.114:0/625970962 <== mon.1 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7ff4d0001ad0 con 0x7ff4d813f5a0 2024-03-29T11:24:33.054 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.062+0000 7ff4ceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:33.054 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.062+0000 7ff4ceffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:24:33.055 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.062+0000 7ff4ceffd640 10 monclient: dump: 2024-03-29T11:24:33.055 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:24:33.055 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:24:33.055 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:24:33.055 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:24:33.055 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:33.055 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:24:33.055 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:24:33.055 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:24:33.055 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:24:33.055 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:24:33.055 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.062+0000 7ff4df331640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:33.055 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.062+0000 7ff4df331640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:33.055 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.062+0000 7ff4df331640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:24:33.056 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.062+0000 7ff4df331640 1 -- 172.21.15.114:0/625970962 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4d804f730 con 0x7ff4d813f5a0 2024-03-29T11:24:33.056 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.063+0000 7ff4dcb2c640 4 set_mon_vals no callback set 2024-03-29T11:24:33.057 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.065+0000 7ff4ceffd640 1 -- 172.21.15.114:0/625970962 <== mon.1 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1594292461 0 0) 0x7ff4d0019820 con 0x7ff4d813f5a0 2024-03-29T11:24:33.057 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.066+0000 7ff4ceffd640 1 -- 172.21.15.114:0/625970962 <== mon.1 v1:172.21.15.176:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3238987313 0 0) 0x7ff4d0053480 con 0x7ff4d813f5a0 2024-03-29T11:24:33.057 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.066+0000 7ff4ceffd640 10 client.4360.objecter ms_dispatch 0x7ff4d8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:24:33.057 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.066+0000 7ff4ceffd640 3 client.4360.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:24:33.057 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.066+0000 7ff4ceffd640 3 client.4360.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:24:33.057 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.066+0000 7ff4ceffd640 20 client.4360.objecter dump_active .. 0 homeless 2024-03-29T11:24:33.057 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.067+0000 7ff4de32f640 10 monclient: get_auth_request con 0x7ff4bc06e000 auth_method 0 2024-03-29T11:24:33.057 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.067+0000 7ff4ceffd640 10 client.4360.objecter ms_handle_connect 0x7ff4bc06e000 2024-03-29T11:24:33.058 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.076+0000 7ff4ceffd640 1 -- 172.21.15.114:0/625970962 <== mon.1 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff4d004f720 con 0x7ff4d813f5a0 2024-03-29T11:24:33.058 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.076+0000 7ff4ceffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:33.058 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.076+0000 7ff4ceffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:33.383 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.402+0000 7ff4df331640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:33.383 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.402+0000 7ff4df331640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:33.383 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.402+0000 7ff4df331640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:24:33.383 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.402+0000 7ff4df331640 1 -- 172.21.15.114:0/625970962 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff4d80b14a0 con 0x7ff4d813f5a0 2024-03-29T11:24:33.384 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.403+0000 7ff4ceffd640 1 -- 172.21.15.114:0/625970962 <== mon.1 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2858224613) 0x7ff4d004f8c0 con 0x7ff4d813f5a0 2024-03-29T11:24:33.384 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.403+0000 7ff4ceffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:33.384 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.403+0000 7ff4ceffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:33.384 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T11:24:33.385 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"b0de2b9b-c45a-44c7-b6ee-e83b39a1e108","created":"2024-03-29T11:20:49.420660+0000","modified":"2024-03-29T11:20:51.700167+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:51.700167+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a64785d3-6b11-499a-b706-cc827ab96a74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5882dadb-e7a8-4604-ba9a-418013c106c8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d7e8c409-bc66-4e4a-bbdb-420bad31388e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0682989-3595-4cd6-90a5-ab9111c278e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"67068120-e3ff-4817-bd06-44feb7479a35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09e8b-8a85-4997-bdea-75d754e6ee67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2f0d9ead-f47f-4cb3-92a4-0a5ad23cd831","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fdb6a25f-6bd8-4c65-9b9b-b460ba7673a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:24:33.388 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.407+0000 7ff4df331640 10 librados: watch_flush enter 2024-03-29T11:24:33.388 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.407+0000 7ff4df331640 10 librados: watch_flush exit 2024-03-29T11:24:33.388 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.407+0000 7ff4df331640 20 client.4360.objecter shutdown clearing up homeless session... 2024-03-29T11:24:33.388 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.407+0000 7ff4df331640 10 client.4360.objecter successfully canceled tick 2024-03-29T11:24:33.388 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.407+0000 7ff4df331640 1 -- 172.21.15.114:0/625970962 >> v1:172.21.15.114:6816/4038755055 conn(0x7ff4bc06e000 legacy=0x7ff4bc0704c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:33.388 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.407+0000 7ff4df331640 10 monclient: shutdown 2024-03-29T11:24:33.388 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.407+0000 7ff4df331640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:33.389 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.407+0000 7ff4df331640 1 -- 172.21.15.114:0/625970962 >> v1:172.21.15.176:6789/0 conn(0x7ff4d813f5a0 legacy=0x7ff4d8133de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:33.389 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.408+0000 7ff4df331640 1 -- 172.21.15.114:0/625970962 shutdown_connections 2024-03-29T11:24:33.389 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.408+0000 7ff4df331640 1 -- 172.21.15.114:0/625970962 >> 172.21.15.114:0/625970962 conn(0x7ff4d806d240 msgr2=0x7ff4d80b44b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:33.389 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.408+0000 7ff4df331640 1 -- 172.21.15.114:0/625970962 shutdown_connections 2024-03-29T11:24:33.389 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.408+0000 7ff4df331640 1 -- 172.21.15.114:0/625970962 wait complete. 2024-03-29T11:24:33.389 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:33.408+0000 7ff4df331640 1 librados: shutdown 2024-03-29T11:24:33.408 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:24:40.214 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:24:40.365 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.383+0000 7fa411b17640 10 monclient: get_monmap_and_config 2024-03-29T11:24:40.365 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.383+0000 7fa411b17640 10 monclient: build_initial_monmap 2024-03-29T11:24:40.365 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.383+0000 7fa411b17640 10 monclient: monmap: 2024-03-29T11:24:40.366 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:24:40.366 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:24:40.366 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:24:40.384435+0000 2024-03-29T11:24:40.366 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:24:40.384435+0000 2024-03-29T11:24:40.366 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:40.366 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:24:40.366 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:24:40.366 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T11:24:40.366 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T11:24:40.366 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:24:40.368 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.386+0000 7fa411b17640 1 Processor -- start 2024-03-29T11:24:40.369 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.387+0000 7fa411b17640 1 -- start start 2024-03-29T11:24:40.369 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.387+0000 7fa411b17640 10 monclient: init 2024-03-29T11:24:40.371 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.388+0000 7fa411b17640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:40.371 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.388+0000 7fa411b17640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:24:40.371 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.388+0000 7fa411b17640 10 monclient(hunting): picked mon.noname-b con 0x7fa404006e30 addr v1:172.21.15.114:6790/0 2024-03-29T11:24:40.371 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.388+0000 7fa411b17640 10 monclient(hunting): picked mon.noname-c con 0x7fa404035e00 addr v1:172.21.15.176:6789/0 2024-03-29T11:24:40.371 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.388+0000 7fa411b17640 10 monclient(hunting): picked mon.noname-a con 0x7fa4040c9670 addr v1:172.21.15.114:6789/0 2024-03-29T11:24:40.371 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.388+0000 7fa411b17640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa404035970 con 0x7fa4040c9670 2024-03-29T11:24:40.371 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.388+0000 7fa411b17640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa404036280 con 0x7fa404035e00 2024-03-29T11:24:40.371 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.388+0000 7fa411b17640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa404035640 con 0x7fa404006e30 2024-03-29T11:24:40.371 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.388+0000 7fa411b17640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:40.371 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.388+0000 7fa411b17640 10 monclient(hunting): authenticate will time out at 1150.570254s 2024-03-29T11:24:40.373 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.391+0000 7fa411316640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fa4040c9670 0x7fa4040d7560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:35683/0 (socket says 172.21.15.114:35683) 2024-03-29T11:24:40.373 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.391+0000 7fa411316640 1 -- 172.21.15.114:0/1446169554 learned_addr learned my addr 172.21.15.114:0/1446169554 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:24:40.374 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.392+0000 7fa40b7fe640 1 -- 172.21.15.114:0/1446169554 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 833563984 0 0) 0x7fa404035970 con 0x7fa4040c9670 2024-03-29T11:24:40.375 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.392+0000 7fa40b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:24:40.375 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.392+0000 7fa40b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:40.375 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.392+0000 7fa40b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:40.375 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.392+0000 7fa40b7fe640 10 monclient(hunting): my global_id is 4359 2024-03-29T11:24:40.375 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.392+0000 7fa40b7fe640 1 -- 172.21.15.114:0/1446169554 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3fc003520 con 0x7fa4040c9670 2024-03-29T11:24:40.375 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.392+0000 7fa40b7fe640 1 -- 172.21.15.114:0/1446169554 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3968347857 0 0) 0x7fa3fc003520 con 0x7fa4040c9670 2024-03-29T11:24:40.375 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.392+0000 7fa40b7fe640 1 -- 172.21.15.114:0/1446169554 >> v1:172.21.15.114:6790/0 conn(0x7fa404006e30 legacy=0x7fa4040cbcc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:24:40.376 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.393+0000 7fa40b7fe640 1 -- 172.21.15.114:0/1446169554 >> v1:172.21.15.176:6789/0 conn(0x7fa404035e00 legacy=0x7fa4040c8f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:24:40.376 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.393+0000 7fa40b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:24:40.376 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.393+0000 7fa40b7fe640 1 monclient: found mon.noname-a 2024-03-29T11:24:40.376 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.393+0000 7fa40b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:40.376 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.393+0000 7fa40b7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.114:6789/0 2024-03-29T11:24:40.376 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.393+0000 7fa40b7fe640 1 -- 172.21.15.114:0/1446169554 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa404034e20 con 0x7fa4040c9670 2024-03-29T11:24:40.376 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.393+0000 7fa40b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:24:40.376 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.393+0000 7fa40b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:24:40.376 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.393+0000 7fa40b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:40.376 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.393+0000 7fa40b7fe640 1 -- 172.21.15.114:0/1446169554 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fa40c04f730 con 0x7fa4040c9670 2024-03-29T11:24:40.377 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.393+0000 7fa40b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:40.377 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.393+0000 7fa40b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:24:40.377 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.393+0000 7fa40b7fe640 10 monclient: dump: 2024-03-29T11:24:40.377 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:24:40.377 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:24:40.377 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:24:40.377 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:24:40.377 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:40.377 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:24:40.377 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:24:40.377 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:24:40.377 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:24:40.377 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:24:40.377 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.393+0000 7fa411b17640 5 monclient: authenticate success, global_id 4359 2024-03-29T11:24:40.377 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.393+0000 7fa411b17640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:40.379 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.394+0000 7fa40b7fe640 1 -- 172.21.15.114:0/1446169554 <== mon.0 v1:172.21.15.114:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa40c0008d0 con 0x7fa4040c9670 2024-03-29T11:24:40.379 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.394+0000 7fa40b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:40.379 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.394+0000 7fa40b7fe640 1 -- 172.21.15.114:0/1446169554 <== mon.0 v1:172.21.15.114:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fa40c0758b0 con 0x7fa4040c9670 2024-03-29T11:24:40.379 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.394+0000 7fa40b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:40.379 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.394+0000 7fa40b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:24:40.379 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.394+0000 7fa40b7fe640 10 monclient: dump: 2024-03-29T11:24:40.379 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:24:40.379 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:24:40.379 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:24:40.379 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:24:40.379 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:40.380 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:24:40.380 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:24:40.380 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:24:40.380 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:24:40.380 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:24:40.380 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.394+0000 7fa411b17640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:40.380 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.394+0000 7fa411b17640 4 set_mon_vals no callback set 2024-03-29T11:24:40.380 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.395+0000 7fa411b17640 10 monclient: shutdown 2024-03-29T11:24:40.380 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.395+0000 7fa411b17640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:40.380 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.395+0000 7fa411b17640 1 -- 172.21.15.114:0/1446169554 >> v1:172.21.15.114:6789/0 conn(0x7fa4040c9670 legacy=0x7fa4040d7560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:40.380 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.396+0000 7fa411b17640 1 -- 172.21.15.114:0/1446169554 shutdown_connections 2024-03-29T11:24:40.380 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.396+0000 7fa411b17640 1 -- 172.21.15.114:0/1446169554 >> 172.21.15.114:0/1446169554 conn(0x7fa404005c00 msgr2=0x7fa404006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:40.380 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.396+0000 7fa411b17640 1 -- 172.21.15.114:0/1446169554 shutdown_connections 2024-03-29T11:24:40.380 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.396+0000 7fa411b17640 1 -- 172.21.15.114:0/1446169554 wait complete. 2024-03-29T11:24:40.380 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.397+0000 7fa411b17640 10 monclient: build_initial_monmap 2024-03-29T11:24:40.381 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.397+0000 7fa411b17640 10 monclient: monmap: 2024-03-29T11:24:40.381 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:24:40.381 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:40.381 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T11:24:40.381 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T11:24:40.381 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:40.381 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:24:40.381 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:24:40.381 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T11:24:40.381 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T11:24:40.382 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:24:40.383 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.397+0000 7fa411b17640 1 librados: starting msgr at 2024-03-29T11:24:40.383 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.397+0000 7fa411b17640 1 librados: starting objecter 2024-03-29T11:24:40.383 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.397+0000 7fa411b17640 1 Processor -- start 2024-03-29T11:24:40.384 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.397+0000 7fa411b17640 1 -- start start 2024-03-29T11:24:40.384 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.397+0000 7fa411b17640 1 librados: setting wanted keys 2024-03-29T11:24:40.384 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.397+0000 7fa411b17640 1 librados: calling monclient init 2024-03-29T11:24:40.384 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.397+0000 7fa411b17640 10 monclient: init 2024-03-29T11:24:40.384 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.398+0000 7fa411b17640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:40.384 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.398+0000 7fa411b17640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:24:40.384 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.398+0000 7fa411b17640 10 monclient(hunting): picked mon.noname-a con 0x7fa404006e30 addr v1:172.21.15.114:6789/0 2024-03-29T11:24:40.384 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.398+0000 7fa411b17640 10 monclient(hunting): picked mon.noname-b con 0x7fa404035e00 addr v1:172.21.15.176:6789/0 2024-03-29T11:24:40.384 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.400+0000 7fa411b17640 10 monclient(hunting): picked mon.noname-c con 0x7fa4040c9670 addr v1:172.21.15.114:6790/0 2024-03-29T11:24:40.385 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.400+0000 7fa411b17640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4040c13f0 con 0x7fa404006e30 2024-03-29T11:24:40.385 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.400+0000 7fa411b17640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa40402d8f0 con 0x7fa404035e00 2024-03-29T11:24:40.385 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.400+0000 7fa411b17640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa40402dbd0 con 0x7fa4040c9670 2024-03-29T11:24:40.385 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.400+0000 7fa411b17640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:40.385 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.400+0000 7fa411b17640 10 monclient(hunting): authenticate will time out at 1150.582840s 2024-03-29T11:24:40.388 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.405+0000 7fa411316640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7fa4040c9670 0x7fa4040c0ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:56357/0 (socket says 172.21.15.114:56357) 2024-03-29T11:24:40.389 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.405+0000 7fa411316640 1 -- 172.21.15.114:0/3193649560 learned_addr learned my addr 172.21.15.114:0/3193649560 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:24:40.389 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.405+0000 7fa4097fa640 10 client.?.objecter ms_handle_connect 0x7fa4040c9670 2024-03-29T11:24:40.389 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.405+0000 7fa4097fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:40.389 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.406+0000 7fa4097fa640 10 client.?.objecter ms_handle_connect 0x7fa404035e00 2024-03-29T11:24:40.389 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.406+0000 7fa4097fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:40.389 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.406+0000 7fa4097fa640 1 -- 172.21.15.114:0/3193649560 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1993650520 0 0) 0x7fa40402dbd0 con 0x7fa4040c9670 2024-03-29T11:24:40.389 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.406+0000 7fa4097fa640 10 monclient(hunting): _negotiate 2024-03-29T11:24:40.389 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.406+0000 7fa4097fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:40.390 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.406+0000 7fa4097fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:40.390 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.406+0000 7fa4097fa640 10 monclient(hunting): my global_id is 4373 2024-03-29T11:24:40.390 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.406+0000 7fa4097fa640 1 -- 172.21.15.114:0/3193649560 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3f0004eb0 con 0x7fa4040c9670 2024-03-29T11:24:40.390 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.406+0000 7fa4097fa640 10 client.?.objecter ms_handle_connect 0x7fa404006e30 2024-03-29T11:24:40.390 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.406+0000 7fa4097fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:40.390 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.406+0000 7fa4097fa640 1 -- 172.21.15.114:0/3193649560 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3514865840 0 0) 0x7fa40402d8f0 con 0x7fa404035e00 2024-03-29T11:24:40.390 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.406+0000 7fa4097fa640 10 monclient(hunting): _negotiate 2024-03-29T11:24:40.390 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.406+0000 7fa4097fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:40.391 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.406+0000 7fa4097fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:40.391 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.406+0000 7fa4097fa640 10 monclient(hunting): my global_id is 4363 2024-03-29T11:24:40.391 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.406+0000 7fa4097fa640 1 -- 172.21.15.114:0/3193649560 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa40402dbd0 con 0x7fa404035e00 2024-03-29T11:24:40.391 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.406+0000 7fa4097fa640 1 -- 172.21.15.114:0/3193649560 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 646575101 0 0) 0x7fa40402dbd0 con 0x7fa404035e00 2024-03-29T11:24:40.391 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.406+0000 7fa4097fa640 1 -- 172.21.15.114:0/3193649560 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa40402d8f0 con 0x7fa404035e00 2024-03-29T11:24:40.392 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.406+0000 7fa4097fa640 1 -- 172.21.15.114:0/3193649560 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fa400001870 con 0x7fa404035e00 2024-03-29T11:24:40.392 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.406+0000 7fa4097fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:40.392 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.407+0000 7fa4097fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:24:40.392 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.407+0000 7fa4097fa640 10 monclient(hunting): dump: 2024-03-29T11:24:40.392 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:24:40.392 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:24:40.392 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:24:40.392 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:24:40.392 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:40.393 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:24:40.393 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:24:40.393 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:24:40.393 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:24:40.393 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:24:40.393 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.407+0000 7fa4097fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:40.393 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.407+0000 7fa4097fa640 1 -- 172.21.15.114:0/3193649560 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 82103541 0 0) 0x7fa3f0004eb0 con 0x7fa4040c9670 2024-03-29T11:24:40.393 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.407+0000 7fa4097fa640 1 -- 172.21.15.114:0/3193649560 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa40402dbd0 con 0x7fa4040c9670 2024-03-29T11:24:40.394 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.407+0000 7fa4097fa640 1 -- 172.21.15.114:0/3193649560 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fa40c04f400 con 0x7fa4040c9670 2024-03-29T11:24:40.394 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.407+0000 7fa4097fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:40.394 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.407+0000 7fa4097fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:40.394 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.407+0000 7fa4097fa640 10 monclient(hunting): dump: 2024-03-29T11:24:40.394 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:24:40.394 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:24:40.394 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:24:40.394 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:24:40.394 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:40.395 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:24:40.395 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:24:40.395 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:24:40.395 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:24:40.395 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:24:40.395 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.407+0000 7fa4097fa640 1 -- 172.21.15.114:0/3193649560 <== mon.1 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1213422369 0 0) 0x7fa40402d8f0 con 0x7fa404035e00 2024-03-29T11:24:40.395 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.407+0000 7fa4097fa640 1 -- 172.21.15.114:0/3193649560 >> v1:172.21.15.114:6790/0 conn(0x7fa4040c9670 legacy=0x7fa4040c0ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:40.395 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.407+0000 7fa4097fa640 1 -- 172.21.15.114:0/3193649560 >> v1:172.21.15.114:6789/0 conn(0x7fa404006e30 legacy=0x7fa404033440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:40.395 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.407+0000 7fa4097fa640 10 monclient: _finish_hunting 0 2024-03-29T11:24:40.395 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.407+0000 7fa4097fa640 1 monclient: found mon.b 2024-03-29T11:24:40.395 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.407+0000 7fa4097fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:40.395 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.407+0000 7fa4097fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:24:40.395 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.407+0000 7fa4097fa640 1 -- 172.21.15.114:0/3193649560 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa40402deb0 con 0x7fa404035e00 2024-03-29T11:24:40.395 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.407+0000 7fa4097fa640 10 monclient: _finish_auth 0 2024-03-29T11:24:40.395 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.407+0000 7fa4097fa640 10 monclient: _check_auth_tickets 2024-03-29T11:24:40.396 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.407+0000 7fa4097fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:40.396 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.407+0000 7fa411b17640 5 monclient: authenticate success, global_id 4363 2024-03-29T11:24:40.396 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.407+0000 7fa411b17640 10 monclient: _renew_subs 2024-03-29T11:24:40.396 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.407+0000 7fa411b17640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:24:40.396 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.407+0000 7fa411b17640 1 -- 172.21.15.114:0/3193649560 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa404164400 con 0x7fa404035e00 2024-03-29T11:24:40.396 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.407+0000 7fa411b17640 10 client.4363.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:40.396 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.407+0000 7fa411b17640 10 monclient: _renew_subs 2024-03-29T11:24:40.396 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.407+0000 7fa411b17640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:24:40.396 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.407+0000 7fa411b17640 1 -- 172.21.15.114:0/3193649560 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa404033d30 con 0x7fa404035e00 2024-03-29T11:24:40.396 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.407+0000 7fa411b17640 1 librados: init done 2024-03-29T11:24:40.398 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.410+0000 7fa4097fa640 1 -- 172.21.15.114:0/3193649560 <== mon.1 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa4000035f0 con 0x7fa404035e00 2024-03-29T11:24:40.398 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.410+0000 7fa4097fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:40.398 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.410+0000 7fa4097fa640 1 -- 172.21.15.114:0/3193649560 <== mon.1 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fa400003900 con 0x7fa404035e00 2024-03-29T11:24:40.398 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.410+0000 7fa4097fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:40.398 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.410+0000 7fa4097fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:24:40.398 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.410+0000 7fa4097fa640 10 monclient: dump: 2024-03-29T11:24:40.398 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:24:40.399 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:24:40.399 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:24:40.399 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:24:40.399 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:40.399 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:24:40.399 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:24:40.399 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:24:40.399 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:24:40.399 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:24:40.399 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.411+0000 7fa40a7fc640 4 set_mon_vals no callback set 2024-03-29T11:24:40.399 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.413+0000 7fa4097fa640 1 -- 172.21.15.114:0/3193649560 <== mon.1 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1594292461 0 0) 0x7fa4000198d0 con 0x7fa404035e00 2024-03-29T11:24:40.399 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.413+0000 7fa411b17640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:40.399 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.413+0000 7fa411b17640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:40.399 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.413+0000 7fa411b17640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:24:40.399 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.413+0000 7fa411b17640 1 -- 172.21.15.114:0/3193649560 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3ec005080 con 0x7fa404035e00 2024-03-29T11:24:40.400 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.413+0000 7fa4097fa640 1 -- 172.21.15.114:0/3193649560 <== mon.1 v1:172.21.15.176:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3238987313 0 0) 0x7fa400083860 con 0x7fa404035e00 2024-03-29T11:24:40.400 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.413+0000 7fa4097fa640 10 client.4363.objecter ms_dispatch 0x7fa40c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:24:40.400 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.413+0000 7fa4097fa640 3 client.4363.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:24:40.400 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.413+0000 7fa4097fa640 3 client.4363.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:24:40.400 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.414+0000 7fa4097fa640 20 client.4363.objecter dump_active .. 0 homeless 2024-03-29T11:24:40.400 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.414+0000 7fa410b15640 10 monclient: get_auth_request con 0x7fa3f006de50 auth_method 0 2024-03-29T11:24:40.400 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.414+0000 7fa4097fa640 10 client.4363.objecter ms_handle_connect 0x7fa3f006de50 2024-03-29T11:24:40.405 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.423+0000 7fa4097fa640 1 -- 172.21.15.114:0/3193649560 <== mon.1 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa40004ef70 con 0x7fa404035e00 2024-03-29T11:24:40.405 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.423+0000 7fa4097fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:40.405 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.423+0000 7fa4097fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:40.729 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.745+0000 7fa411b17640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:40.730 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.745+0000 7fa411b17640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:40.730 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.745+0000 7fa411b17640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:24:40.730 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.745+0000 7fa411b17640 1 -- 172.21.15.114:0/3193649560 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa3ec005330 con 0x7fa404035e00 2024-03-29T11:24:40.730 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.747+0000 7fa4097fa640 1 -- 172.21.15.114:0/3193649560 <== mon.1 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2858224613) 0x7fa400051070 con 0x7fa404035e00 2024-03-29T11:24:40.730 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.747+0000 7fa4097fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:40.730 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.747+0000 7fa4097fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:40.731 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T11:24:40.731 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"b0de2b9b-c45a-44c7-b6ee-e83b39a1e108","created":"2024-03-29T11:20:49.420660+0000","modified":"2024-03-29T11:20:51.700167+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:51.700167+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a64785d3-6b11-499a-b706-cc827ab96a74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5882dadb-e7a8-4604-ba9a-418013c106c8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d7e8c409-bc66-4e4a-bbdb-420bad31388e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0682989-3595-4cd6-90a5-ab9111c278e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"67068120-e3ff-4817-bd06-44feb7479a35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09e8b-8a85-4997-bdea-75d754e6ee67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2f0d9ead-f47f-4cb3-92a4-0a5ad23cd831","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fdb6a25f-6bd8-4c65-9b9b-b460ba7673a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:24:40.735 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.751+0000 7fa411b17640 10 librados: watch_flush enter 2024-03-29T11:24:40.735 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.751+0000 7fa411b17640 10 librados: watch_flush exit 2024-03-29T11:24:40.735 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.752+0000 7fa411b17640 20 client.4363.objecter shutdown clearing up homeless session... 2024-03-29T11:24:40.736 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.752+0000 7fa411b17640 10 client.4363.objecter successfully canceled tick 2024-03-29T11:24:40.736 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.752+0000 7fa411b17640 1 -- 172.21.15.114:0/3193649560 >> v1:172.21.15.114:6816/4038755055 conn(0x7fa3f006de50 legacy=0x7fa3f0070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:40.736 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.752+0000 7fa411b17640 10 monclient: shutdown 2024-03-29T11:24:40.736 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.752+0000 7fa411b17640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:40.736 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.752+0000 7fa411b17640 1 -- 172.21.15.114:0/3193649560 >> v1:172.21.15.176:6789/0 conn(0x7fa404035e00 legacy=0x7fa4040c05d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:40.737 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.754+0000 7fa411b17640 1 -- 172.21.15.114:0/3193649560 shutdown_connections 2024-03-29T11:24:40.737 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.755+0000 7fa411b17640 1 -- 172.21.15.114:0/3193649560 >> 172.21.15.114:0/3193649560 conn(0x7fa404005c00 msgr2=0x7fa4040c8d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:40.739 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.757+0000 7fa411b17640 1 -- 172.21.15.114:0/3193649560 shutdown_connections 2024-03-29T11:24:40.740 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.757+0000 7fa411b17640 1 -- 172.21.15.114:0/3193649560 wait complete. 2024-03-29T11:24:40.740 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:40.757+0000 7fa411b17640 1 librados: shutdown 2024-03-29T11:24:40.761 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:24:47.571 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:24:47.727 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.743+0000 7fdd6c97b640 10 monclient: get_monmap_and_config 2024-03-29T11:24:47.727 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.743+0000 7fdd6c97b640 10 monclient: build_initial_monmap 2024-03-29T11:24:47.727 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.743+0000 7fdd6c97b640 10 monclient: monmap: 2024-03-29T11:24:47.727 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:24:47.727 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:24:47.727 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:24:47.744103+0000 2024-03-29T11:24:47.727 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:24:47.744103+0000 2024-03-29T11:24:47.727 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:47.727 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:24:47.727 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:24:47.727 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T11:24:47.727 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T11:24:47.728 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:24:47.728 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.744+0000 7fdd6c97b640 1 Processor -- start 2024-03-29T11:24:47.728 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.744+0000 7fdd6c97b640 1 -- start start 2024-03-29T11:24:47.728 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.744+0000 7fdd6c97b640 10 monclient: init 2024-03-29T11:24:47.729 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.745+0000 7fdd6c97b640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:47.729 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.745+0000 7fdd6c97b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:24:47.729 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.745+0000 7fdd6c97b640 10 monclient(hunting): picked mon.noname-b con 0x7fdd600e2d10 addr v1:172.21.15.114:6790/0 2024-03-29T11:24:47.729 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.745+0000 7fdd6c97b640 10 monclient(hunting): picked mon.noname-c con 0x7fdd6002b090 addr v1:172.21.15.176:6789/0 2024-03-29T11:24:47.729 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.745+0000 7fdd6c97b640 10 monclient(hunting): picked mon.noname-a con 0x7fdd60006940 addr v1:172.21.15.114:6789/0 2024-03-29T11:24:47.729 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.745+0000 7fdd6c97b640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd6002ac40 con 0x7fdd60006940 2024-03-29T11:24:47.729 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.745+0000 7fdd6c97b640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd60006dd0 con 0x7fdd6002b090 2024-03-29T11:24:47.729 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.745+0000 7fdd6c97b640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd6002a430 con 0x7fdd600e2d10 2024-03-29T11:24:47.730 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.745+0000 7fdd6c97b640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:47.730 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.745+0000 7fdd6c97b640 10 monclient(hunting): authenticate will time out at 1157.927500s 2024-03-29T11:24:47.735 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.751+0000 7fdd6759e640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fdd60006940 0x7fdd600f0ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:46207/0 (socket says 172.21.15.114:46207) 2024-03-29T11:24:47.735 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.751+0000 7fdd6759e640 1 -- 172.21.15.114:0/863716996 learned_addr learned my addr 172.21.15.114:0/863716996 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:24:47.735 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.752+0000 7fdd65d9b640 1 -- 172.21.15.114:0/863716996 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3022825895 0 0) 0x7fdd6002ac40 con 0x7fdd60006940 2024-03-29T11:24:47.735 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.752+0000 7fdd65d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:24:47.735 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.752+0000 7fdd65d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:47.735 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.752+0000 7fdd65d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:47.736 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.752+0000 7fdd65d9b640 10 monclient(hunting): my global_id is 4365 2024-03-29T11:24:47.736 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.752+0000 7fdd65d9b640 1 -- 172.21.15.114:0/863716996 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd50003520 con 0x7fdd60006940 2024-03-29T11:24:47.736 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.752+0000 7fdd65d9b640 1 -- 172.21.15.114:0/863716996 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3088747071 0 0) 0x7fdd6002a430 con 0x7fdd600e2d10 2024-03-29T11:24:47.736 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.752+0000 7fdd65d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:24:47.736 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.752+0000 7fdd65d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:47.736 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.752+0000 7fdd65d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:47.736 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.752+0000 7fdd65d9b640 10 monclient(hunting): my global_id is 4376 2024-03-29T11:24:47.736 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.752+0000 7fdd65d9b640 1 -- 172.21.15.114:0/863716996 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd500048a0 con 0x7fdd600e2d10 2024-03-29T11:24:47.736 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.752+0000 7fdd65d9b640 1 -- 172.21.15.114:0/863716996 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2382850122 0 0) 0x7fdd60006dd0 con 0x7fdd6002b090 2024-03-29T11:24:47.736 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.752+0000 7fdd65d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:24:47.736 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.752+0000 7fdd65d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:47.736 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.752+0000 7fdd65d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:47.736 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.752+0000 7fdd65d9b640 10 monclient(hunting): my global_id is 4366 2024-03-29T11:24:47.737 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.752+0000 7fdd65d9b640 1 -- 172.21.15.114:0/863716996 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd6002a430 con 0x7fdd6002b090 2024-03-29T11:24:47.737 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.752+0000 7fdd65d9b640 1 -- 172.21.15.114:0/863716996 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2721011934 0 0) 0x7fdd50003520 con 0x7fdd60006940 2024-03-29T11:24:47.737 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.752+0000 7fdd65d9b640 1 -- 172.21.15.114:0/863716996 >> v1:172.21.15.114:6790/0 conn(0x7fdd600e2d10 legacy=0x7fdd600e8900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:47.737 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.752+0000 7fdd65d9b640 1 -- 172.21.15.114:0/863716996 >> v1:172.21.15.176:6789/0 conn(0x7fdd6002b090 legacy=0x7fdd60029c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:47.737 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.752+0000 7fdd65d9b640 10 monclient: _finish_hunting 0 2024-03-29T11:24:47.737 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.752+0000 7fdd65d9b640 1 monclient: found mon.noname-a 2024-03-29T11:24:47.737 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.752+0000 7fdd65d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:47.737 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.752+0000 7fdd65d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.114:6789/0 2024-03-29T11:24:47.737 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.752+0000 7fdd65d9b640 1 -- 172.21.15.114:0/863716996 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd6002a5c0 con 0x7fdd60006940 2024-03-29T11:24:47.737 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.752+0000 7fdd65d9b640 10 monclient: _finish_auth 0 2024-03-29T11:24:47.737 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.752+0000 7fdd65d9b640 10 monclient: _check_auth_tickets 2024-03-29T11:24:47.737 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.752+0000 7fdd65d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:47.737 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.752+0000 7fdd65d9b640 1 -- 172.21.15.114:0/863716996 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fdd6804f730 con 0x7fdd60006940 2024-03-29T11:24:47.737 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.752+0000 7fdd65d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:47.738 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.752+0000 7fdd65d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:24:47.738 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.752+0000 7fdd65d9b640 10 monclient: dump: 2024-03-29T11:24:47.738 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:24:47.738 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:24:47.738 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:24:47.738 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:24:47.738 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:47.738 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:24:47.738 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:24:47.738 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:24:47.738 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:24:47.738 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:24:47.738 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.752+0000 7fdd65d9b640 1 -- 172.21.15.114:0/863716996 <== mon.0 v1:172.21.15.114:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdd68075f10 con 0x7fdd60006940 2024-03-29T11:24:47.738 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.752+0000 7fdd65d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:47.739 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.752+0000 7fdd65d9b640 1 -- 172.21.15.114:0/863716996 <== mon.0 v1:172.21.15.114:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fdd6804f730 con 0x7fdd60006940 2024-03-29T11:24:47.739 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.752+0000 7fdd65d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:47.739 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.752+0000 7fdd65d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:24:47.739 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.752+0000 7fdd65d9b640 10 monclient: dump: 2024-03-29T11:24:47.739 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:24:47.739 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:24:47.739 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:24:47.739 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:24:47.739 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:47.739 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:24:47.739 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:24:47.739 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:24:47.739 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:24:47.739 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:24:47.739 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.752+0000 7fdd6c97b640 5 monclient: authenticate success, global_id 4365 2024-03-29T11:24:47.740 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.752+0000 7fdd6c97b640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:47.740 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.752+0000 7fdd6c97b640 4 set_mon_vals no callback set 2024-03-29T11:24:47.740 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.753+0000 7fdd6c97b640 10 monclient: shutdown 2024-03-29T11:24:47.740 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.753+0000 7fdd6c97b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:47.740 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.753+0000 7fdd6c97b640 1 -- 172.21.15.114:0/863716996 >> v1:172.21.15.114:6789/0 conn(0x7fdd60006940 legacy=0x7fdd600f0ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:47.740 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.758+0000 7fdd6c97b640 1 -- 172.21.15.114:0/863716996 shutdown_connections 2024-03-29T11:24:47.741 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.758+0000 7fdd6c97b640 1 -- 172.21.15.114:0/863716996 >> 172.21.15.114:0/863716996 conn(0x7fdd600058f0 msgr2=0x7fdd60005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:47.741 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.759+0000 7fdd6c97b640 1 -- 172.21.15.114:0/863716996 shutdown_connections 2024-03-29T11:24:47.743 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.761+0000 7fdd6c97b640 1 -- 172.21.15.114:0/863716996 wait complete. 2024-03-29T11:24:47.744 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.762+0000 7fdd6c97b640 10 monclient: build_initial_monmap 2024-03-29T11:24:47.744 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.762+0000 7fdd6c97b640 10 monclient: monmap: 2024-03-29T11:24:47.744 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:24:47.745 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:47.745 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T11:24:47.745 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T11:24:47.745 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:47.745 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:24:47.745 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:24:47.745 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T11:24:47.745 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T11:24:47.745 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:24:47.750 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.762+0000 7fdd6c97b640 1 librados: starting msgr at 2024-03-29T11:24:47.750 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.762+0000 7fdd6c97b640 1 librados: starting objecter 2024-03-29T11:24:47.750 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.763+0000 7fdd6c97b640 1 Processor -- start 2024-03-29T11:24:47.750 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.763+0000 7fdd6c97b640 1 -- start start 2024-03-29T11:24:47.750 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.763+0000 7fdd6c97b640 1 librados: setting wanted keys 2024-03-29T11:24:47.750 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.763+0000 7fdd6c97b640 1 librados: calling monclient init 2024-03-29T11:24:47.750 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.763+0000 7fdd6c97b640 10 monclient: init 2024-03-29T11:24:47.750 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.764+0000 7fdd6c97b640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:47.750 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.764+0000 7fdd6c97b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:24:47.750 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.764+0000 7fdd6c97b640 10 monclient(hunting): picked mon.noname-b con 0x7fdd60006940 addr v1:172.21.15.176:6789/0 2024-03-29T11:24:47.750 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.764+0000 7fdd6c97b640 10 monclient(hunting): picked mon.noname-a con 0x7fdd6002b090 addr v1:172.21.15.114:6789/0 2024-03-29T11:24:47.750 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.764+0000 7fdd6c97b640 10 monclient(hunting): picked mon.noname-c con 0x7fdd600e2d10 addr v1:172.21.15.114:6790/0 2024-03-29T11:24:47.751 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.764+0000 7fdd6c97b640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd600dbe70 con 0x7fdd6002b090 2024-03-29T11:24:47.751 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.764+0000 7fdd6c97b640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd600e80e0 con 0x7fdd60006940 2024-03-29T11:24:47.751 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.764+0000 7fdd6c97b640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd600e83c0 con 0x7fdd600e2d10 2024-03-29T11:24:47.751 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.764+0000 7fdd6c97b640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:47.751 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.764+0000 7fdd6c97b640 10 monclient(hunting): authenticate will time out at 1157.946572s 2024-03-29T11:24:47.752 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.770+0000 7fdd6759e640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7fdd600e2d10 0x7fdd60033e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:37513/0 (socket says 172.21.15.114:37513) 2024-03-29T11:24:47.755 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.771+0000 7fdd66d9d640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7fdd60006940 0x7fdd600e6ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.114:55251/0 (socket says 172.21.15.114:55251) 2024-03-29T11:24:47.755 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.771+0000 7fdd66d9d640 1 -- 172.21.15.114:0/2727045708 learned_addr learned my addr 172.21.15.114:0/2727045708 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:24:47.755 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.771+0000 7fdd4f7fe640 10 client.?.objecter ms_handle_connect 0x7fdd600e2d10 2024-03-29T11:24:47.755 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.771+0000 7fdd4f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:47.755 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.771+0000 7fdd4f7fe640 10 client.?.objecter ms_handle_connect 0x7fdd60006940 2024-03-29T11:24:47.755 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.771+0000 7fdd4f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:47.755 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.771+0000 7fdd4f7fe640 10 client.?.objecter ms_handle_connect 0x7fdd6002b090 2024-03-29T11:24:47.755 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.771+0000 7fdd4f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:47.755 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.771+0000 7fdd4f7fe640 1 -- 172.21.15.114:0/2727045708 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 22854702 0 0) 0x7fdd600e83c0 con 0x7fdd600e2d10 2024-03-29T11:24:47.755 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.771+0000 7fdd4f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:24:47.755 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.771+0000 7fdd4f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:47.755 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.771+0000 7fdd4f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:47.755 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.771+0000 7fdd4f7fe640 10 monclient(hunting): my global_id is 4379 2024-03-29T11:24:47.755 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.771+0000 7fdd4f7fe640 1 -- 172.21.15.114:0/2727045708 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd54004eb0 con 0x7fdd600e2d10 2024-03-29T11:24:47.756 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.771+0000 7fdd4f7fe640 1 -- 172.21.15.114:0/2727045708 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3735271071 0 0) 0x7fdd600e80e0 con 0x7fdd60006940 2024-03-29T11:24:47.756 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.772+0000 7fdd4f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:24:47.756 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.772+0000 7fdd4f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:47.756 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.772+0000 7fdd4f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:47.756 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.772+0000 7fdd4f7fe640 10 monclient(hunting): my global_id is 4369 2024-03-29T11:24:47.756 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.772+0000 7fdd4f7fe640 1 -- 172.21.15.114:0/2727045708 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd600e83c0 con 0x7fdd60006940 2024-03-29T11:24:47.756 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.772+0000 7fdd4f7fe640 1 -- 172.21.15.114:0/2727045708 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1742813566 0 0) 0x7fdd600dbe70 con 0x7fdd6002b090 2024-03-29T11:24:47.756 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.772+0000 7fdd4f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:24:47.756 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.772+0000 7fdd4f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:47.756 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.772+0000 7fdd4f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:47.756 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.772+0000 7fdd4f7fe640 10 monclient(hunting): my global_id is 4368 2024-03-29T11:24:47.756 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.772+0000 7fdd4f7fe640 1 -- 172.21.15.114:0/2727045708 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd600e80e0 con 0x7fdd6002b090 2024-03-29T11:24:47.761 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.775+0000 7fdd4f7fe640 1 -- 172.21.15.114:0/2727045708 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2281572104 0 0) 0x7fdd54004eb0 con 0x7fdd600e2d10 2024-03-29T11:24:47.761 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.775+0000 7fdd4f7fe640 1 -- 172.21.15.114:0/2727045708 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdd600dbe70 con 0x7fdd600e2d10 2024-03-29T11:24:47.761 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.775+0000 7fdd4f7fe640 1 -- 172.21.15.114:0/2727045708 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3992100198 0 0) 0x7fdd600e83c0 con 0x7fdd60006940 2024-03-29T11:24:47.761 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.775+0000 7fdd4f7fe640 1 -- 172.21.15.114:0/2727045708 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdd54004eb0 con 0x7fdd60006940 2024-03-29T11:24:47.761 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.775+0000 7fdd4f7fe640 1 -- 172.21.15.114:0/2727045708 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3063942286 0 0) 0x7fdd600e80e0 con 0x7fdd6002b090 2024-03-29T11:24:47.761 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.775+0000 7fdd4f7fe640 1 -- 172.21.15.114:0/2727045708 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdd600e83c0 con 0x7fdd6002b090 2024-03-29T11:24:47.762 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.775+0000 7fdd4f7fe640 1 -- 172.21.15.114:0/2727045708 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fdd6804f730 con 0x7fdd600e2d10 2024-03-29T11:24:47.762 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.775+0000 7fdd4f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:47.762 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.775+0000 7fdd4f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:24:47.762 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.775+0000 7fdd4f7fe640 10 monclient(hunting): dump: 2024-03-29T11:24:47.762 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:24:47.762 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:24:47.762 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:24:47.762 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:24:47.762 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:47.762 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:24:47.763 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:24:47.763 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:24:47.763 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:24:47.763 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:24:47.763 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.775+0000 7fdd4f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:47.763 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.775+0000 7fdd4f7fe640 1 -- 172.21.15.114:0/2727045708 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fdd5c001830 con 0x7fdd60006940 2024-03-29T11:24:47.763 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.775+0000 7fdd4f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:47.763 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.775+0000 7fdd4f7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:24:47.763 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.775+0000 7fdd4f7fe640 10 monclient(hunting): dump: 2024-03-29T11:24:47.763 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:24:47.763 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:24:47.763 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:24:47.763 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:24:47.763 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:47.763 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:24:47.764 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:24:47.764 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:24:47.764 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:24:47.764 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:24:47.764 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.775+0000 7fdd4f7fe640 1 -- 172.21.15.114:0/2727045708 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fdd580018b0 con 0x7fdd6002b090 2024-03-29T11:24:47.764 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.775+0000 7fdd4f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:47.764 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.775+0000 7fdd4f7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:24:47.764 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.775+0000 7fdd4f7fe640 10 monclient(hunting): dump: 2024-03-29T11:24:47.764 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:24:47.764 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:24:47.764 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:24:47.764 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:24:47.764 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:47.764 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:24:47.765 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:24:47.765 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:24:47.765 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:24:47.765 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:24:47.765 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.775+0000 7fdd4f7fe640 1 -- 172.21.15.114:0/2727045708 <== mon.2 v1:172.21.15.114:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3823435397 0 0) 0x7fdd600dbe70 con 0x7fdd600e2d10 2024-03-29T11:24:47.765 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.775+0000 7fdd4f7fe640 1 -- 172.21.15.114:0/2727045708 >> v1:172.21.15.176:6789/0 conn(0x7fdd60006940 legacy=0x7fdd600e6ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:47.765 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.776+0000 7fdd4f7fe640 1 -- 172.21.15.114:0/2727045708 >> v1:172.21.15.114:6789/0 conn(0x7fdd6002b090 legacy=0x7fdd60033770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:47.765 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.776+0000 7fdd4f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:24:47.765 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.776+0000 7fdd4f7fe640 1 monclient: found mon.c 2024-03-29T11:24:47.765 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.776+0000 7fdd4f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:47.765 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.776+0000 7fdd4f7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:24:47.765 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.776+0000 7fdd4f7fe640 1 -- 172.21.15.114:0/2727045708 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd600dc110 con 0x7fdd600e2d10 2024-03-29T11:24:47.765 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.776+0000 7fdd4f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:24:47.765 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.776+0000 7fdd4f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:24:47.765 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.776+0000 7fdd4f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:47.766 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.776+0000 7fdd4f7fe640 1 -- 172.21.15.114:0/2727045708 <== mon.2 v1:172.21.15.114:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdd68075710 con 0x7fdd600e2d10 2024-03-29T11:24:47.766 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.776+0000 7fdd4f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:47.766 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.776+0000 7fdd4f7fe640 1 -- 172.21.15.114:0/2727045708 <== mon.2 v1:172.21.15.114:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fdd68067d90 con 0x7fdd600e2d10 2024-03-29T11:24:47.766 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.776+0000 7fdd4f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:47.766 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.776+0000 7fdd4f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:47.766 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.776+0000 7fdd4f7fe640 10 monclient: dump: 2024-03-29T11:24:47.766 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:24:47.766 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:24:47.766 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:24:47.766 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:24:47.766 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:47.766 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:24:47.766 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:24:47.766 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:24:47.766 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:24:47.767 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:24:47.767 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.776+0000 7fdd6559a640 4 set_mon_vals no callback set 2024-03-29T11:24:47.767 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.777+0000 7fdd6c97b640 5 monclient: authenticate success, global_id 4379 2024-03-29T11:24:47.767 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.777+0000 7fdd6c97b640 10 monclient: _renew_subs 2024-03-29T11:24:47.767 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.777+0000 7fdd6c97b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:24:47.767 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.777+0000 7fdd6c97b640 1 -- 172.21.15.114:0/2727045708 --> v1:172.21.15.114:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd60034590 con 0x7fdd600e2d10 2024-03-29T11:24:47.767 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.778+0000 7fdd6c97b640 10 client.4379.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:47.767 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.778+0000 7fdd6c97b640 10 monclient: _renew_subs 2024-03-29T11:24:47.767 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.778+0000 7fdd6c97b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:24:47.767 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.778+0000 7fdd6c97b640 1 -- 172.21.15.114:0/2727045708 --> v1:172.21.15.114:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd600d6840 con 0x7fdd600e2d10 2024-03-29T11:24:47.767 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.778+0000 7fdd6c97b640 1 librados: init done 2024-03-29T11:24:47.768 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.780+0000 7fdd4f7fe640 1 -- 172.21.15.114:0/2727045708 <== mon.2 v1:172.21.15.114:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1594292461 0 0) 0x7fdd68067ef0 con 0x7fdd600e2d10 2024-03-29T11:24:47.768 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.781+0000 7fdd6c97b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:47.768 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.781+0000 7fdd6c97b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:47.768 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.781+0000 7fdd6c97b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:24:47.768 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.781+0000 7fdd6c97b640 1 -- 172.21.15.114:0/2727045708 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd6002b620 con 0x7fdd600e2d10 2024-03-29T11:24:47.768 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.784+0000 7fdd66d9d640 10 monclient: get_auth_request con 0x7fdd5406e160 auth_method 0 2024-03-29T11:24:47.768 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.784+0000 7fdd4f7fe640 10 client.4379.objecter ms_handle_connect 0x7fdd5406e160 2024-03-29T11:24:47.769 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.784+0000 7fdd4f7fe640 1 -- 172.21.15.114:0/2727045708 <== mon.2 v1:172.21.15.114:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3238987313 0 0) 0x7fdd68075880 con 0x7fdd600e2d10 2024-03-29T11:24:47.769 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.784+0000 7fdd4f7fe640 10 client.4379.objecter ms_dispatch 0x7fdd68000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:24:47.769 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.784+0000 7fdd4f7fe640 3 client.4379.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:24:47.769 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.784+0000 7fdd4f7fe640 3 client.4379.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:24:47.769 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.785+0000 7fdd4f7fe640 20 client.4379.objecter dump_active .. 0 homeless 2024-03-29T11:24:47.775 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.791+0000 7fdd4f7fe640 1 -- 172.21.15.114:0/2727045708 <== mon.2 v1:172.21.15.114:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdd680ba430 con 0x7fdd600e2d10 2024-03-29T11:24:47.775 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.791+0000 7fdd4f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:47.776 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:47.791+0000 7fdd4f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:48.099 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:48.115+0000 7fdd6c97b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:48.099 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:48.115+0000 7fdd6c97b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:48.099 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:48.115+0000 7fdd6c97b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:24:48.099 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:48.115+0000 7fdd6c97b640 1 -- 172.21.15.114:0/2727045708 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdd600dc2a0 con 0x7fdd600e2d10 2024-03-29T11:24:48.105 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T11:24:48.105 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"b0de2b9b-c45a-44c7-b6ee-e83b39a1e108","created":"2024-03-29T11:20:49.420660+0000","modified":"2024-03-29T11:20:51.700167+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:51.700167+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a64785d3-6b11-499a-b706-cc827ab96a74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5882dadb-e7a8-4604-ba9a-418013c106c8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d7e8c409-bc66-4e4a-bbdb-420bad31388e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0682989-3595-4cd6-90a5-ab9111c278e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"67068120-e3ff-4817-bd06-44feb7479a35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09e8b-8a85-4997-bdea-75d754e6ee67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2f0d9ead-f47f-4cb3-92a4-0a5ad23cd831","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fdb6a25f-6bd8-4c65-9b9b-b460ba7673a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:24:48.105 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:48.119+0000 7fdd4f7fe640 1 -- 172.21.15.114:0/2727045708 <== mon.2 v1:172.21.15.114:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2858224613) 0x7fdd680ba5d0 con 0x7fdd600e2d10 2024-03-29T11:24:48.105 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:48.119+0000 7fdd4f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:48.105 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:48.119+0000 7fdd4f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:48.107 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:48.124+0000 7fdd6c97b640 10 librados: watch_flush enter 2024-03-29T11:24:48.107 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:48.124+0000 7fdd6c97b640 10 librados: watch_flush exit 2024-03-29T11:24:48.107 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:48.124+0000 7fdd6c97b640 20 client.4379.objecter shutdown clearing up homeless session... 2024-03-29T11:24:48.107 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:48.124+0000 7fdd6c97b640 10 client.4379.objecter successfully canceled tick 2024-03-29T11:24:48.107 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:48.124+0000 7fdd6c97b640 1 -- 172.21.15.114:0/2727045708 >> v1:172.21.15.114:6816/4038755055 conn(0x7fdd5406e160 legacy=0x7fdd54070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:48.107 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:48.124+0000 7fdd6c97b640 10 monclient: shutdown 2024-03-29T11:24:48.107 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:48.124+0000 7fdd6c97b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:48.107 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:48.124+0000 7fdd6c97b640 1 -- 172.21.15.114:0/2727045708 >> v1:172.21.15.114:6790/0 conn(0x7fdd600e2d10 legacy=0x7fdd60033e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:48.110 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:48.127+0000 7fdd6c97b640 1 -- 172.21.15.114:0/2727045708 shutdown_connections 2024-03-29T11:24:48.110 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:48.127+0000 7fdd6c97b640 1 -- 172.21.15.114:0/2727045708 >> 172.21.15.114:0/2727045708 conn(0x7fdd600058f0 msgr2=0x7fdd600e4a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:48.110 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:48.127+0000 7fdd6c97b640 1 -- 172.21.15.114:0/2727045708 shutdown_connections 2024-03-29T11:24:48.110 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:48.127+0000 7fdd6c97b640 1 -- 172.21.15.114:0/2727045708 wait complete. 2024-03-29T11:24:48.110 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:48.127+0000 7fdd6c97b640 1 librados: shutdown 2024-03-29T11:24:48.128 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:24:54.938 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:24:55.088 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.104+0000 7f4190a47640 10 monclient: get_monmap_and_config 2024-03-29T11:24:55.088 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.104+0000 7f4190a47640 10 monclient: build_initial_monmap 2024-03-29T11:24:55.088 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.104+0000 7f4190a47640 10 monclient: monmap: 2024-03-29T11:24:55.088 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:24:55.088 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:24:55.088 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:24:55.105978+0000 2024-03-29T11:24:55.088 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:24:55.105978+0000 2024-03-29T11:24:55.088 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:55.088 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:24:55.088 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:24:55.089 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T11:24:55.089 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T11:24:55.089 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:24:55.092 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.106+0000 7f4190a47640 1 Processor -- start 2024-03-29T11:24:55.092 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.106+0000 7f4190a47640 1 -- start start 2024-03-29T11:24:55.092 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.106+0000 7f4190a47640 10 monclient: init 2024-03-29T11:24:55.092 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.107+0000 7f4190a47640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:55.092 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.107+0000 7f4190a47640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:24:55.092 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.107+0000 7f4190a47640 10 monclient(hunting): picked mon.noname-c con 0x7f418c0b0860 addr v1:172.21.15.176:6789/0 2024-03-29T11:24:55.092 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.107+0000 7f4190a47640 10 monclient(hunting): picked mon.noname-a con 0x7f418c13ef30 addr v1:172.21.15.114:6789/0 2024-03-29T11:24:55.095 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.110+0000 7f4190a47640 10 monclient(hunting): picked mon.noname-b con 0x7f418c0af8e0 addr v1:172.21.15.114:6790/0 2024-03-29T11:24:55.095 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.111+0000 7f4190a47640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f418c0b0c40 con 0x7f418c13ef30 2024-03-29T11:24:55.095 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.111+0000 7f4190a47640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f418c0aed10 con 0x7f418c0b0860 2024-03-29T11:24:55.095 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.111+0000 7f4190a47640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f418c0afd50 con 0x7f418c0af8e0 2024-03-29T11:24:55.096 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.111+0000 7f4190a47640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:55.096 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.111+0000 7f4190a47640 10 monclient(hunting): authenticate will time out at 1165.293628s 2024-03-29T11:24:55.096 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.111+0000 7f418a59c640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f418c0b0860 0x7f418c13e820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.114:60871/0 (socket says 172.21.15.114:60871) 2024-03-29T11:24:55.096 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.111+0000 7f418a59c640 1 -- 172.21.15.114:0/28422341 learned_addr learned my addr 172.21.15.114:0/28422341 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:24:55.096 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.111+0000 7f418959a640 1 -- 172.21.15.114:0/28422341 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3225533838 0 0) 0x7f418c0aed10 con 0x7f418c0b0860 2024-03-29T11:24:55.096 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.112+0000 7f418959a640 10 monclient(hunting): _negotiate 2024-03-29T11:24:55.096 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.112+0000 7f418959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:55.096 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.112+0000 7f418959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:55.096 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.112+0000 7f418959a640 10 monclient(hunting): my global_id is 4372 2024-03-29T11:24:55.096 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.112+0000 7f418959a640 1 -- 172.21.15.114:0/28422341 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4174003520 con 0x7f418c0b0860 2024-03-29T11:24:55.097 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.113+0000 7f418959a640 1 -- 172.21.15.114:0/28422341 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1695357390 0 0) 0x7f418c0afd50 con 0x7f418c0af8e0 2024-03-29T11:24:55.097 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.113+0000 7f418959a640 10 monclient(hunting): _negotiate 2024-03-29T11:24:55.097 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.113+0000 7f418959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:55.097 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.113+0000 7f418959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:55.097 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.113+0000 7f418959a640 10 monclient(hunting): my global_id is 4382 2024-03-29T11:24:55.097 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.113+0000 7f418959a640 1 -- 172.21.15.114:0/28422341 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f418c0aed10 con 0x7f418c0af8e0 2024-03-29T11:24:55.097 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.114+0000 7f418959a640 1 -- 172.21.15.114:0/28422341 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3797595660 0 0) 0x7f4174003520 con 0x7f418c0b0860 2024-03-29T11:24:55.097 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.114+0000 7f418959a640 1 -- 172.21.15.114:0/28422341 >> v1:172.21.15.114:6790/0 conn(0x7f418c0af8e0 legacy=0x7f418c149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:55.097 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.114+0000 7f418959a640 1 -- 172.21.15.114:0/28422341 >> v1:172.21.15.114:6789/0 conn(0x7f418c13ef30 legacy=0x7f418c0af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:24:55.098 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.114+0000 7f418959a640 10 monclient: _finish_hunting 0 2024-03-29T11:24:55.098 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.114+0000 7f418959a640 1 monclient: found mon.noname-c 2024-03-29T11:24:55.098 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.114+0000 7f418959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:55.098 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.114+0000 7f418959a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.176:6789/0 2024-03-29T11:24:55.098 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.114+0000 7f418959a640 1 -- 172.21.15.114:0/28422341 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f418c0ab1b0 con 0x7f418c0b0860 2024-03-29T11:24:55.098 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.114+0000 7f418959a640 10 monclient: _finish_auth 0 2024-03-29T11:24:55.098 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.114+0000 7f418959a640 10 monclient: _check_auth_tickets 2024-03-29T11:24:55.098 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.114+0000 7f418959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:55.098 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.114+0000 7f418959a640 1 -- 172.21.15.114:0/28422341 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f4180001690 con 0x7f418c0b0860 2024-03-29T11:24:55.098 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.114+0000 7f418959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:55.098 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.114+0000 7f418959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:24:55.098 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.114+0000 7f418959a640 10 monclient: dump: 2024-03-29T11:24:55.098 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:24:55.099 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:24:55.099 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:24:55.099 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:24:55.099 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:55.099 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:24:55.099 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:24:55.099 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:24:55.099 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:24:55.099 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:24:55.099 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.114+0000 7f418959a640 1 -- 172.21.15.114:0/28422341 <== mon.1 v1:172.21.15.176:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f41800017f0 con 0x7f418c0b0860 2024-03-29T11:24:55.099 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.114+0000 7f418959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:55.100 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.114+0000 7f418959a640 1 -- 172.21.15.114:0/28422341 <== mon.1 v1:172.21.15.176:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f4180001c00 con 0x7f418c0b0860 2024-03-29T11:24:55.100 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.114+0000 7f418959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:55.100 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.114+0000 7f418959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:24:55.100 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.114+0000 7f418959a640 10 monclient: dump: 2024-03-29T11:24:55.100 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:24:55.100 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:24:55.100 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:24:55.100 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:24:55.100 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:55.100 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:24:55.100 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:24:55.100 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:24:55.100 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:24:55.100 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:24:55.101 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.114+0000 7f4190a47640 5 monclient: authenticate success, global_id 4372 2024-03-29T11:24:55.101 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.114+0000 7f4190a47640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:55.101 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.114+0000 7f4190a47640 4 set_mon_vals no callback set 2024-03-29T11:24:55.101 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.115+0000 7f4190a47640 10 monclient: shutdown 2024-03-29T11:24:55.102 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.115+0000 7f4190a47640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:55.102 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.115+0000 7f4190a47640 1 -- 172.21.15.114:0/28422341 >> v1:172.21.15.176:6789/0 conn(0x7f418c0b0860 legacy=0x7f418c13e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:55.102 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.117+0000 7f4190a47640 1 -- 172.21.15.114:0/28422341 shutdown_connections 2024-03-29T11:24:55.102 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.117+0000 7f4190a47640 1 -- 172.21.15.114:0/28422341 >> 172.21.15.114:0/28422341 conn(0x7f418c06b660 msgr2=0x7f418c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:55.102 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.118+0000 7f4190a47640 1 -- 172.21.15.114:0/28422341 shutdown_connections 2024-03-29T11:24:55.106 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.121+0000 7f4190a47640 1 -- 172.21.15.114:0/28422341 wait complete. 2024-03-29T11:24:55.106 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.122+0000 7f4190a47640 10 monclient: build_initial_monmap 2024-03-29T11:24:55.106 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.122+0000 7f4190a47640 10 monclient: monmap: 2024-03-29T11:24:55.106 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:24:55.107 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:55.107 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T11:24:55.107 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T11:24:55.107 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:55.107 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:24:55.107 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:24:55.107 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T11:24:55.107 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T11:24:55.107 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:24:55.107 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.123+0000 7f4190a47640 1 librados: starting msgr at 2024-03-29T11:24:55.107 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.123+0000 7f4190a47640 1 librados: starting objecter 2024-03-29T11:24:55.107 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.123+0000 7f4190a47640 1 Processor -- start 2024-03-29T11:24:55.108 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.124+0000 7f4190a47640 1 -- start start 2024-03-29T11:24:55.108 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.124+0000 7f4190a47640 1 librados: setting wanted keys 2024-03-29T11:24:55.108 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.124+0000 7f4190a47640 1 librados: calling monclient init 2024-03-29T11:24:55.108 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.124+0000 7f4190a47640 10 monclient: init 2024-03-29T11:24:55.111 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.124+0000 7f4190a47640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:55.111 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.124+0000 7f4190a47640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:24:55.111 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.125+0000 7f4190a47640 10 monclient(hunting): picked mon.noname-b con 0x7f418c0af8e0 addr v1:172.21.15.176:6789/0 2024-03-29T11:24:55.111 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.125+0000 7f4190a47640 10 monclient(hunting): picked mon.noname-c con 0x7f418c0b0860 addr v1:172.21.15.114:6790/0 2024-03-29T11:24:55.111 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.127+0000 7f4190a47640 10 monclient(hunting): picked mon.noname-a con 0x7f418c13ef30 addr v1:172.21.15.114:6789/0 2024-03-29T11:24:55.112 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.127+0000 7f4190a47640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f418c136170 con 0x7f418c13ef30 2024-03-29T11:24:55.112 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.127+0000 7f4190a47640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f418c13a7f0 con 0x7f418c0af8e0 2024-03-29T11:24:55.112 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.127+0000 7f4190a47640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f418c13b540 con 0x7f418c0b0860 2024-03-29T11:24:55.112 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.127+0000 7f4190a47640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:55.112 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.127+0000 7f4190a47640 10 monclient(hunting): authenticate will time out at 1165.309671s 2024-03-29T11:24:55.112 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.129+0000 7f418ad9d640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f418c13ef30 0x7f418c1dd8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:43215/0 (socket says 172.21.15.114:43215) 2024-03-29T11:24:55.112 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.129+0000 7f418ad9d640 1 -- 172.21.15.114:0/1326353462 learned_addr learned my addr 172.21.15.114:0/1326353462 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:24:55.113 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.129+0000 7f417affd640 10 client.?.objecter ms_handle_connect 0x7f418c13ef30 2024-03-29T11:24:55.113 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.129+0000 7f417affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:55.113 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.129+0000 7f417affd640 10 client.?.objecter ms_handle_connect 0x7f418c0af8e0 2024-03-29T11:24:55.113 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.129+0000 7f417affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:55.113 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.130+0000 7f417affd640 1 -- 172.21.15.114:0/1326353462 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1170800891 0 0) 0x7f418c13a7f0 con 0x7f418c0af8e0 2024-03-29T11:24:55.113 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.130+0000 7f417affd640 10 monclient(hunting): _negotiate 2024-03-29T11:24:55.113 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.130+0000 7f417affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:55.113 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.130+0000 7f417affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:55.113 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.130+0000 7f417affd640 10 monclient(hunting): my global_id is 4375 2024-03-29T11:24:55.113 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.130+0000 7f417affd640 1 -- 172.21.15.114:0/1326353462 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f416c004eb0 con 0x7f418c0af8e0 2024-03-29T11:24:55.116 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.130+0000 7f417affd640 1 -- 172.21.15.114:0/1326353462 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3348922229 0 0) 0x7f416c004eb0 con 0x7f418c0af8e0 2024-03-29T11:24:55.116 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.130+0000 7f417affd640 1 -- 172.21.15.114:0/1326353462 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f418c13a7f0 con 0x7f418c0af8e0 2024-03-29T11:24:55.116 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.130+0000 7f417affd640 1 -- 172.21.15.114:0/1326353462 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f4180001b00 con 0x7f418c0af8e0 2024-03-29T11:24:55.116 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.130+0000 7f417affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:55.117 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.130+0000 7f417affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:24:55.117 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.130+0000 7f417affd640 10 monclient(hunting): dump: 2024-03-29T11:24:55.117 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:24:55.117 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:24:55.117 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:24:55.117 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:24:55.117 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:55.117 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:24:55.117 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:24:55.117 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:24:55.117 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:24:55.117 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:24:55.117 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.130+0000 7f417affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:55.117 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.130+0000 7f417affd640 1 -- 172.21.15.114:0/1326353462 <== mon.1 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1088862223 0 0) 0x7f418c13a7f0 con 0x7f418c0af8e0 2024-03-29T11:24:55.118 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.130+0000 7f417affd640 1 -- 172.21.15.114:0/1326353462 >> v1:172.21.15.114:6790/0 conn(0x7f418c0b0860 legacy=0x7f418c1d3210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:55.118 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.131+0000 7f417affd640 1 -- 172.21.15.114:0/1326353462 >> v1:172.21.15.114:6789/0 conn(0x7f418c13ef30 legacy=0x7f418c1dd8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:55.118 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.131+0000 7f417affd640 10 monclient: _finish_hunting 0 2024-03-29T11:24:55.118 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.131+0000 7f417affd640 1 monclient: found mon.b 2024-03-29T11:24:55.118 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.131+0000 7f417affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:55.118 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.131+0000 7f417affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:24:55.118 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.131+0000 7f417affd640 1 -- 172.21.15.114:0/1326353462 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f418c13b820 con 0x7f418c0af8e0 2024-03-29T11:24:55.118 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.131+0000 7f417affd640 10 monclient: _finish_auth 0 2024-03-29T11:24:55.118 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.131+0000 7f417affd640 10 monclient: _check_auth_tickets 2024-03-29T11:24:55.118 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.131+0000 7f417affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:55.118 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.131+0000 7f417affd640 10 client.?.objecter ms_handle_connect 0x7f418c0b0860 2024-03-29T11:24:55.118 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.131+0000 7f417affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:55.118 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.131+0000 7f4190a47640 5 monclient: authenticate success, global_id 4375 2024-03-29T11:24:55.118 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.131+0000 7f4190a47640 10 monclient: _renew_subs 2024-03-29T11:24:55.119 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.131+0000 7f4190a47640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:24:55.119 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.131+0000 7f4190a47640 1 -- 172.21.15.114:0/1326353462 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f418c13a240 con 0x7f418c0af8e0 2024-03-29T11:24:55.119 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.131+0000 7f4190a47640 10 client.4375.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:55.119 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.131+0000 7f4190a47640 10 monclient: _renew_subs 2024-03-29T11:24:55.119 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.131+0000 7f4190a47640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:24:55.119 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.131+0000 7f4190a47640 1 -- 172.21.15.114:0/1326353462 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f418c1d3b00 con 0x7f418c0af8e0 2024-03-29T11:24:55.119 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.131+0000 7f4190a47640 1 librados: init done 2024-03-29T11:24:55.119 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.132+0000 7f4178ff9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:55.119 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.132+0000 7f4178ff9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:55.119 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.132+0000 7f4178ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:24:55.119 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.132+0000 7f4178ff9640 1 -- 172.21.15.114:0/1326353462 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4170005080 con 0x7f418c0af8e0 2024-03-29T11:24:55.119 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.133+0000 7f417affd640 1 -- 172.21.15.114:0/1326353462 <== mon.1 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4180004b00 con 0x7f418c0af8e0 2024-03-29T11:24:55.119 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.133+0000 7f417affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:55.119 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.133+0000 7f417affd640 1 -- 172.21.15.114:0/1326353462 <== mon.1 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f4180004f10 con 0x7f418c0af8e0 2024-03-29T11:24:55.120 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.133+0000 7f417affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:55.120 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.133+0000 7f4188d99640 4 set_mon_vals no callback set 2024-03-29T11:24:55.120 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.133+0000 7f417affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:24:55.120 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.133+0000 7f417affd640 10 monclient: dump: 2024-03-29T11:24:55.120 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:24:55.120 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:24:55.120 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:24:55.120 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:24:55.120 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:55.120 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:24:55.120 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:24:55.120 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:24:55.120 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:24:55.120 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:24:55.121 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.134+0000 7f417affd640 1 -- 172.21.15.114:0/1326353462 <== mon.1 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1594292461 0 0) 0x7f4180005130 con 0x7f418c0af8e0 2024-03-29T11:24:55.121 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.134+0000 7f417affd640 1 -- 172.21.15.114:0/1326353462 <== mon.1 v1:172.21.15.176:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3238987313 0 0) 0x7f4180084ca0 con 0x7f418c0af8e0 2024-03-29T11:24:55.121 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.134+0000 7f417affd640 10 client.4375.objecter ms_dispatch 0x7f418c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:24:55.121 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.134+0000 7f417affd640 3 client.4375.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:24:55.121 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.134+0000 7f417affd640 3 client.4375.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:24:55.122 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.135+0000 7f417affd640 20 client.4375.objecter dump_active .. 0 homeless 2024-03-29T11:24:55.122 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.135+0000 7f4189d9b640 10 monclient: get_auth_request con 0x7f416c06dc20 auth_method 0 2024-03-29T11:24:55.122 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.135+0000 7f417affd640 10 client.4375.objecter ms_handle_connect 0x7f416c06dc20 2024-03-29T11:24:55.125 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.142+0000 7f417affd640 1 -- 172.21.15.114:0/1326353462 <== mon.1 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4180050700 con 0x7f418c0af8e0 2024-03-29T11:24:55.125 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.142+0000 7f417affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:55.125 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.142+0000 7f417affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:55.446 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.463+0000 7f4178ff9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:55.446 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.463+0000 7f4178ff9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:55.446 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.463+0000 7f4178ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:24:55.447 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.463+0000 7f4178ff9640 1 -- 172.21.15.114:0/1326353462 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4170005330 con 0x7f418c0af8e0 2024-03-29T11:24:55.447 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.464+0000 7f417affd640 1 -- 172.21.15.114:0/1326353462 <== mon.1 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2858224613) 0x7f4180052570 con 0x7f418c0af8e0 2024-03-29T11:24:55.447 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.464+0000 7f417affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:55.447 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.464+0000 7f417affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:55.447 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T11:24:55.448 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"b0de2b9b-c45a-44c7-b6ee-e83b39a1e108","created":"2024-03-29T11:20:49.420660+0000","modified":"2024-03-29T11:20:51.700167+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:51.700167+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a64785d3-6b11-499a-b706-cc827ab96a74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5882dadb-e7a8-4604-ba9a-418013c106c8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d7e8c409-bc66-4e4a-bbdb-420bad31388e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0682989-3595-4cd6-90a5-ab9111c278e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"67068120-e3ff-4817-bd06-44feb7479a35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09e8b-8a85-4997-bdea-75d754e6ee67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2f0d9ead-f47f-4cb3-92a4-0a5ad23cd831","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fdb6a25f-6bd8-4c65-9b9b-b460ba7673a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:24:55.450 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.468+0000 7f4178ff9640 10 librados: watch_flush enter 2024-03-29T11:24:55.451 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.468+0000 7f4178ff9640 10 librados: watch_flush exit 2024-03-29T11:24:55.451 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.468+0000 7f4178ff9640 20 client.4375.objecter shutdown clearing up homeless session... 2024-03-29T11:24:55.451 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.468+0000 7f4178ff9640 10 client.4375.objecter successfully canceled tick 2024-03-29T11:24:55.451 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.468+0000 7f4178ff9640 1 -- 172.21.15.114:0/1326353462 >> v1:172.21.15.114:6816/4038755055 conn(0x7f416c06dc20 legacy=0x7f416c0700e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:55.452 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.468+0000 7f4178ff9640 10 monclient: shutdown 2024-03-29T11:24:55.452 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.468+0000 7f4178ff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:55.452 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.468+0000 7f4178ff9640 1 -- 172.21.15.114:0/1326353462 >> v1:172.21.15.176:6789/0 conn(0x7f418c0af8e0 legacy=0x7f418c139b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:55.452 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.469+0000 7f4178ff9640 1 -- 172.21.15.114:0/1326353462 shutdown_connections 2024-03-29T11:24:55.452 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.469+0000 7f4178ff9640 1 -- 172.21.15.114:0/1326353462 >> 172.21.15.114:0/1326353462 conn(0x7f418c06b660 msgr2=0x7f418c1c8f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:55.454 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.472+0000 7f4178ff9640 1 -- 172.21.15.114:0/1326353462 shutdown_connections 2024-03-29T11:24:55.454 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.472+0000 7f4178ff9640 1 -- 172.21.15.114:0/1326353462 wait complete. 2024-03-29T11:24:55.455 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:24:55.472+0000 7f4178ff9640 1 librados: shutdown 2024-03-29T11:24:55.474 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:25:02.281 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:25:02.434 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.450+0000 7f31ef0c8640 10 monclient: get_monmap_and_config 2024-03-29T11:25:02.435 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.450+0000 7f31ef0c8640 10 monclient: build_initial_monmap 2024-03-29T11:25:02.435 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.450+0000 7f31ef0c8640 10 monclient: monmap: 2024-03-29T11:25:02.435 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:25:02.435 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:25:02.435 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:25:02.451429+0000 2024-03-29T11:25:02.435 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:25:02.451429+0000 2024-03-29T11:25:02.435 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:02.435 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:25:02.435 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:25:02.435 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T11:25:02.435 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T11:25:02.435 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:25:02.436 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.451+0000 7f31ef0c8640 1 Processor -- start 2024-03-29T11:25:02.436 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.451+0000 7f31ef0c8640 1 -- start start 2024-03-29T11:25:02.436 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.451+0000 7f31ef0c8640 10 monclient: init 2024-03-29T11:25:02.436 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.452+0000 7f31ef0c8640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:02.436 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.452+0000 7f31ef0c8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:25:02.447 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.453+0000 7f31ef0c8640 10 monclient(hunting): picked mon.noname-c con 0x7f31e80a9ac0 addr v1:172.21.15.176:6789/0 2024-03-29T11:25:02.447 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.453+0000 7f31ef0c8640 10 monclient(hunting): picked mon.noname-a con 0x7f31e813ee90 addr v1:172.21.15.114:6789/0 2024-03-29T11:25:02.448 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.453+0000 7f31ef0c8640 10 monclient(hunting): picked mon.noname-b con 0x7f31e80af900 addr v1:172.21.15.114:6790/0 2024-03-29T11:25:02.448 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.453+0000 7f31ef0c8640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31e80afd10 con 0x7f31e813ee90 2024-03-29T11:25:02.448 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.453+0000 7f31ef0c8640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31e80af6f0 con 0x7f31e80a9ac0 2024-03-29T11:25:02.448 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.453+0000 7f31ef0c8640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31e80aac80 con 0x7f31e80af900 2024-03-29T11:25:02.448 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.453+0000 7f31ef0c8640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:02.448 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.453+0000 7f31ef0c8640 10 monclient(hunting): authenticate will time out at 1172.635739s 2024-03-29T11:25:02.448 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.456+0000 7f31e7fff640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f31e813ee90 0x7f31e813b5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:59677/0 (socket says 172.21.15.114:59677) 2024-03-29T11:25:02.449 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.457+0000 7f31e7fff640 1 -- 172.21.15.114:0/2998440497 learned_addr learned my addr 172.21.15.114:0/2998440497 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:25:02.449 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.457+0000 7f31e77fe640 1 -- 172.21.15.114:0/2998440497 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2143797704 0 0) 0x7f31e80afd10 con 0x7f31e813ee90 2024-03-29T11:25:02.449 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.458+0000 7f31e77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:25:02.449 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.458+0000 7f31e77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:02.449 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.458+0000 7f31e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:02.449 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.458+0000 7f31e77fe640 10 monclient(hunting): my global_id is 4374 2024-03-29T11:25:02.449 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.458+0000 7f31e77fe640 1 -- 172.21.15.114:0/2998440497 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31d0003520 con 0x7f31e813ee90 2024-03-29T11:25:02.449 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.458+0000 7f31e77fe640 1 -- 172.21.15.114:0/2998440497 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 315934069 0 0) 0x7f31e80aac80 con 0x7f31e80af900 2024-03-29T11:25:02.450 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.458+0000 7f31e77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:25:02.450 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.458+0000 7f31e77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:02.450 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.458+0000 7f31e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:02.450 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.458+0000 7f31e77fe640 10 monclient(hunting): my global_id is 4388 2024-03-29T11:25:02.450 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.458+0000 7f31e77fe640 1 -- 172.21.15.114:0/2998440497 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31e80afd10 con 0x7f31e80af900 2024-03-29T11:25:02.450 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.459+0000 7f31e77fe640 1 -- 172.21.15.114:0/2998440497 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2820274704 0 0) 0x7f31d0003520 con 0x7f31e813ee90 2024-03-29T11:25:02.450 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.459+0000 7f31e77fe640 1 -- 172.21.15.114:0/2998440497 >> v1:172.21.15.114:6790/0 conn(0x7f31e80af900 legacy=0x7f31e8149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:02.450 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.459+0000 7f31e77fe640 1 -- 172.21.15.114:0/2998440497 >> v1:172.21.15.176:6789/0 conn(0x7f31e80a9ac0 legacy=0x7f31e80adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:25:02.450 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.459+0000 7f31e77fe640 10 monclient: _finish_hunting 0 2024-03-29T11:25:02.451 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.459+0000 7f31e77fe640 1 monclient: found mon.noname-a 2024-03-29T11:25:02.451 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.459+0000 7f31e77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:02.451 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.459+0000 7f31e77fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.114:6789/0 2024-03-29T11:25:02.451 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.459+0000 7f31e77fe640 1 -- 172.21.15.114:0/2998440497 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31e80aaf60 con 0x7f31e813ee90 2024-03-29T11:25:02.451 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.459+0000 7f31e77fe640 10 monclient: _finish_auth 0 2024-03-29T11:25:02.451 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.459+0000 7f31e77fe640 10 monclient: _check_auth_tickets 2024-03-29T11:25:02.451 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.459+0000 7f31e77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:02.451 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.459+0000 7f31e77fe640 1 -- 172.21.15.114:0/2998440497 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f31d8001690 con 0x7f31e813ee90 2024-03-29T11:25:02.451 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.459+0000 7f31e77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:02.451 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.459+0000 7f31e77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:25:02.452 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.459+0000 7f31e77fe640 10 monclient: dump: 2024-03-29T11:25:02.452 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:25:02.452 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:25:02.452 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:25:02.452 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:25:02.452 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:02.452 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:25:02.452 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:25:02.452 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:25:02.452 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:25:02.452 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:25:02.453 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.459+0000 7f31ef0c8640 5 monclient: authenticate success, global_id 4374 2024-03-29T11:25:02.453 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.459+0000 7f31ef0c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:02.453 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.459+0000 7f31e77fe640 1 -- 172.21.15.114:0/2998440497 <== mon.0 v1:172.21.15.114:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f31d80017f0 con 0x7f31e813ee90 2024-03-29T11:25:02.453 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.460+0000 7f31e77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:02.453 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.460+0000 7f31e77fe640 1 -- 172.21.15.114:0/2998440497 <== mon.0 v1:172.21.15.114:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f31d8001c40 con 0x7f31e813ee90 2024-03-29T11:25:02.453 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.460+0000 7f31e77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:02.453 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.460+0000 7f31e77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:25:02.453 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.460+0000 7f31e77fe640 10 monclient: dump: 2024-03-29T11:25:02.453 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:25:02.453 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:25:02.453 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:25:02.454 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:25:02.454 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:02.454 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:25:02.454 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:25:02.454 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:25:02.454 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:25:02.454 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:25:02.454 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.460+0000 7f31ef0c8640 10 monclient: get_monmap_and_config success 2024-03-29T11:25:02.454 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.460+0000 7f31ef0c8640 4 set_mon_vals no callback set 2024-03-29T11:25:02.454 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.461+0000 7f31ef0c8640 10 monclient: shutdown 2024-03-29T11:25:02.455 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.461+0000 7f31ef0c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:02.455 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.461+0000 7f31ef0c8640 1 -- 172.21.15.114:0/2998440497 >> v1:172.21.15.114:6789/0 conn(0x7f31e813ee90 legacy=0x7f31e813b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:02.455 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.461+0000 7f31ef0c8640 1 -- 172.21.15.114:0/2998440497 shutdown_connections 2024-03-29T11:25:02.455 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.461+0000 7f31ef0c8640 1 -- 172.21.15.114:0/2998440497 >> 172.21.15.114:0/2998440497 conn(0x7f31e80b35e0 msgr2=0x7f31e80b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:02.455 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.461+0000 7f31ef0c8640 1 -- 172.21.15.114:0/2998440497 shutdown_connections 2024-03-29T11:25:02.455 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.461+0000 7f31ef0c8640 1 -- 172.21.15.114:0/2998440497 wait complete. 2024-03-29T11:25:02.455 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.462+0000 7f31ef0c8640 10 monclient: build_initial_monmap 2024-03-29T11:25:02.455 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.462+0000 7f31ef0c8640 10 monclient: monmap: 2024-03-29T11:25:02.455 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:25:02.455 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:25:02.455 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T11:25:02.455 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T11:25:02.455 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:02.455 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:25:02.456 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:25:02.456 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T11:25:02.456 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T11:25:02.456 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:25:02.456 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.463+0000 7f31ef0c8640 1 librados: starting msgr at 2024-03-29T11:25:02.456 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.463+0000 7f31ef0c8640 1 librados: starting objecter 2024-03-29T11:25:02.456 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.463+0000 7f31ef0c8640 1 Processor -- start 2024-03-29T11:25:02.456 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.463+0000 7f31ef0c8640 1 -- start start 2024-03-29T11:25:02.456 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.463+0000 7f31ef0c8640 1 librados: setting wanted keys 2024-03-29T11:25:02.456 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.464+0000 7f31ef0c8640 1 librados: calling monclient init 2024-03-29T11:25:02.457 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.464+0000 7f31ef0c8640 10 monclient: init 2024-03-29T11:25:02.464 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.465+0000 7f31ef0c8640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:02.464 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.465+0000 7f31ef0c8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:25:02.464 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.465+0000 7f31ef0c8640 10 monclient(hunting): picked mon.noname-c con 0x7f31e80a9ac0 addr v1:172.21.15.114:6790/0 2024-03-29T11:25:02.465 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.469+0000 7f31ef0c8640 10 monclient(hunting): picked mon.noname-a con 0x7f31e80af900 addr v1:172.21.15.114:6789/0 2024-03-29T11:25:02.465 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.470+0000 7f31ece64640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f31e80a9ac0 0x7f31e8139860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:33725/0 (socket says 172.21.15.114:33725) 2024-03-29T11:25:02.465 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.472+0000 7f31ef0c8640 10 monclient(hunting): picked mon.noname-b con 0x7f31e813ee90 addr v1:172.21.15.176:6789/0 2024-03-29T11:25:02.465 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.472+0000 7f31ece64640 1 -- 172.21.15.114:0/611364186 learned_addr learned my addr 172.21.15.114:0/611364186 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:25:02.465 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.473+0000 7f31ef0c8640 1 -- 172.21.15.114:0/611364186 --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31e806f490 con 0x7f31e80af900 2024-03-29T11:25:02.465 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.473+0000 7f31ef0c8640 1 -- 172.21.15.114:0/611364186 --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31e806f000 con 0x7f31e813ee90 2024-03-29T11:25:02.465 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.473+0000 7f31ef0c8640 1 -- 172.21.15.114:0/611364186 --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31e8070710 con 0x7f31e80a9ac0 2024-03-29T11:25:02.465 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.473+0000 7f31ef0c8640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:02.465 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.473+0000 7f31ef0c8640 10 monclient(hunting): authenticate will time out at 1172.655661s 2024-03-29T11:25:02.465 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.473+0000 7f31e57fa640 10 client.?.objecter ms_handle_connect 0x7f31e813ee90 2024-03-29T11:25:02.465 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.473+0000 7f31e57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:02.465 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.473+0000 7f31e57fa640 1 -- 172.21.15.114:0/611364186 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 624584963 0 0) 0x7f31e806f000 con 0x7f31e813ee90 2024-03-29T11:25:02.466 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.473+0000 7f31e57fa640 10 monclient(hunting): _negotiate 2024-03-29T11:25:02.466 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.473+0000 7f31e57fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:02.466 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.473+0000 7f31e57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:02.466 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.473+0000 7f31e57fa640 10 monclient(hunting): my global_id is 4378 2024-03-29T11:25:02.466 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.473+0000 7f31e57fa640 1 -- 172.21.15.114:0/611364186 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31d4004eb0 con 0x7f31e813ee90 2024-03-29T11:25:02.466 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.474+0000 7f31e57fa640 10 client.?.objecter ms_handle_connect 0x7f31e80a9ac0 2024-03-29T11:25:02.466 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.474+0000 7f31e57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:02.466 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.474+0000 7f31e57fa640 1 -- 172.21.15.114:0/611364186 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 408964487 0 0) 0x7f31e8070710 con 0x7f31e80a9ac0 2024-03-29T11:25:02.466 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.474+0000 7f31e57fa640 10 monclient(hunting): _negotiate 2024-03-29T11:25:02.466 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.474+0000 7f31e57fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:02.466 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.474+0000 7f31e57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:02.467 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.474+0000 7f31e57fa640 10 monclient(hunting): my global_id is 4391 2024-03-29T11:25:02.467 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.474+0000 7f31e57fa640 1 -- 172.21.15.114:0/611364186 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31e806f000 con 0x7f31e80a9ac0 2024-03-29T11:25:02.467 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.474+0000 7f31e57fa640 1 -- 172.21.15.114:0/611364186 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3003912087 0 0) 0x7f31e806f000 con 0x7f31e80a9ac0 2024-03-29T11:25:02.467 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.474+0000 7f31e57fa640 1 -- 172.21.15.114:0/611364186 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f31d40064f0 con 0x7f31e80a9ac0 2024-03-29T11:25:02.467 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.474+0000 7f31e57fa640 1 -- 172.21.15.114:0/611364186 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f31dc004000 con 0x7f31e80a9ac0 2024-03-29T11:25:02.467 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.474+0000 7f31e57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:02.467 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.474+0000 7f31e57fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:25:02.467 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.474+0000 7f31e57fa640 10 monclient(hunting): dump: 2024-03-29T11:25:02.467 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:25:02.467 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:25:02.467 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:25:02.467 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:25:02.468 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:02.468 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:25:02.468 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:25:02.468 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:25:02.468 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:25:02.468 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:25:02.468 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.475+0000 7f31e57fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:25:02.468 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.475+0000 7f31e57fa640 1 -- 172.21.15.114:0/611364186 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3181758103 0 0) 0x7f31d4004eb0 con 0x7f31e813ee90 2024-03-29T11:25:02.468 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.475+0000 7f31e57fa640 1 -- 172.21.15.114:0/611364186 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f31d4006bc0 con 0x7f31e813ee90 2024-03-29T11:25:02.468 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.475+0000 7f31e57fa640 1 -- 172.21.15.114:0/611364186 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f31e0001a30 con 0x7f31e813ee90 2024-03-29T11:25:02.468 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.475+0000 7f31e57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:02.468 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.475+0000 7f31e57fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:25:02.469 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.475+0000 7f31e57fa640 10 monclient(hunting): dump: 2024-03-29T11:25:02.469 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:25:02.469 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:25:02.469 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:25:02.469 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:25:02.469 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:02.469 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:25:02.469 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:25:02.469 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:25:02.469 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:25:02.469 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:25:02.469 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.477+0000 7f31e57fa640 1 -- 172.21.15.114:0/611364186 <== mon.1 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2113116433 0 0) 0x7f31d4006bc0 con 0x7f31e813ee90 2024-03-29T11:25:02.469 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.477+0000 7f31e57fa640 1 -- 172.21.15.114:0/611364186 >> v1:172.21.15.114:6790/0 conn(0x7f31e80a9ac0 legacy=0x7f31e8139860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:02.469 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.477+0000 7f31e57fa640 1 -- 172.21.15.114:0/611364186 >> v1:172.21.15.114:6789/0 conn(0x7f31e80af900 legacy=0x7f31e81d6680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:02.470 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.477+0000 7f31e57fa640 10 monclient: _finish_hunting 0 2024-03-29T11:25:02.470 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.477+0000 7f31e57fa640 1 monclient: found mon.b 2024-03-29T11:25:02.470 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.477+0000 7f31e57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:02.470 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.477+0000 7f31e57fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:25:02.470 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.477+0000 7f31e57fa640 1 -- 172.21.15.114:0/611364186 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31e806f640 con 0x7f31e813ee90 2024-03-29T11:25:02.470 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.477+0000 7f31e57fa640 10 monclient: _finish_auth 0 2024-03-29T11:25:02.470 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.477+0000 7f31e57fa640 10 monclient: _check_auth_tickets 2024-03-29T11:25:02.470 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.477+0000 7f31e57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:02.470 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.478+0000 7f31ef0c8640 5 monclient: authenticate success, global_id 4378 2024-03-29T11:25:02.470 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.478+0000 7f31ef0c8640 10 monclient: _renew_subs 2024-03-29T11:25:02.470 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.478+0000 7f31ef0c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:25:02.470 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.478+0000 7f31ef0c8640 1 -- 172.21.15.114:0/611364186 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31e81423e0 con 0x7f31e813ee90 2024-03-29T11:25:02.471 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.478+0000 7f31e57fa640 1 -- 172.21.15.114:0/611364186 <== mon.1 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f31e00020e0 con 0x7f31e813ee90 2024-03-29T11:25:02.471 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.478+0000 7f31e57fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:02.471 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.478+0000 7f31e57fa640 1 -- 172.21.15.114:0/611364186 <== mon.1 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f31e0003b10 con 0x7f31e813ee90 2024-03-29T11:25:02.471 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.478+0000 7f31e57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:02.471 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.478+0000 7f31e57fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:25:02.471 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.478+0000 7f31e57fa640 10 monclient: dump: 2024-03-29T11:25:02.471 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:25:02.471 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:25:02.471 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:25:02.471 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:25:02.471 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:02.471 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:25:02.471 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:25:02.472 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:25:02.472 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:25:02.472 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:25:02.472 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.479+0000 7f31ef0c8640 10 client.4378.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:25:02.472 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.479+0000 7f31e67fc640 4 set_mon_vals no callback set 2024-03-29T11:25:02.472 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.479+0000 7f31ef0c8640 10 monclient: _renew_subs 2024-03-29T11:25:02.472 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.479+0000 7f31ef0c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:25:02.472 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.479+0000 7f31ef0c8640 1 -- 172.21.15.114:0/611364186 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f31e8071060 con 0x7f31e813ee90 2024-03-29T11:25:02.472 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.479+0000 7f31ef0c8640 1 librados: init done 2024-03-29T11:25:02.472 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.480+0000 7f31ef0c8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:02.472 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.480+0000 7f31ef0c8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:02.472 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.480+0000 7f31ef0c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:25:02.472 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.480+0000 7f31ef0c8640 1 -- 172.21.15.114:0/611364186 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31e813bcd0 con 0x7f31e813ee90 2024-03-29T11:25:02.473 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.484+0000 7f31e57fa640 1 -- 172.21.15.114:0/611364186 <== mon.1 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1594292461 0 0) 0x7f31e0019b60 con 0x7f31e813ee90 2024-03-29T11:25:02.473 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.485+0000 7f31e57fa640 1 -- 172.21.15.114:0/611364186 <== mon.1 v1:172.21.15.176:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3238987313 0 0) 0x7f31e0083990 con 0x7f31e813ee90 2024-03-29T11:25:02.474 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.485+0000 7f31e57fa640 10 client.4378.objecter ms_dispatch 0x7f31e8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:25:02.474 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.485+0000 7f31e57fa640 3 client.4378.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:25:02.474 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.485+0000 7f31e57fa640 3 client.4378.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:25:02.474 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.485+0000 7f31e57fa640 20 client.4378.objecter dump_active .. 0 homeless 2024-03-29T11:25:02.474 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.485+0000 7f31ece64640 10 monclient: get_auth_request con 0x7f31d406e5b0 auth_method 0 2024-03-29T11:25:02.474 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.485+0000 7f31e57fa640 10 client.4378.objecter ms_handle_connect 0x7f31d406e5b0 2024-03-29T11:25:02.474 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.490+0000 7f31e57fa640 1 -- 172.21.15.114:0/611364186 <== mon.1 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f31e004f0a0 con 0x7f31e813ee90 2024-03-29T11:25:02.474 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.490+0000 7f31e57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:02.474 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.490+0000 7f31e57fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:25:02.799 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.815+0000 7f31ef0c8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:02.799 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.815+0000 7f31ef0c8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:02.799 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.815+0000 7f31ef0c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:25:02.799 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.815+0000 7f31ef0c8640 1 -- 172.21.15.114:0/611364186 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f31e8142670 con 0x7f31e813ee90 2024-03-29T11:25:02.799 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.816+0000 7f31e57fa640 1 -- 172.21.15.114:0/611364186 <== mon.1 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2858224613) 0x7f31e0003970 con 0x7f31e813ee90 2024-03-29T11:25:02.799 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.816+0000 7f31e57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:02.800 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.816+0000 7f31e57fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:25:02.802 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T11:25:02.802 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"b0de2b9b-c45a-44c7-b6ee-e83b39a1e108","created":"2024-03-29T11:20:49.420660+0000","modified":"2024-03-29T11:20:51.700167+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:51.700167+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a64785d3-6b11-499a-b706-cc827ab96a74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5882dadb-e7a8-4604-ba9a-418013c106c8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d7e8c409-bc66-4e4a-bbdb-420bad31388e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0682989-3595-4cd6-90a5-ab9111c278e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"67068120-e3ff-4817-bd06-44feb7479a35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09e8b-8a85-4997-bdea-75d754e6ee67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2f0d9ead-f47f-4cb3-92a4-0a5ad23cd831","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fdb6a25f-6bd8-4c65-9b9b-b460ba7673a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:25:02.805 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.823+0000 7f31ef0c8640 10 librados: watch_flush enter 2024-03-29T11:25:02.806 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.823+0000 7f31ef0c8640 10 librados: watch_flush exit 2024-03-29T11:25:02.806 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.823+0000 7f31ef0c8640 20 client.4378.objecter shutdown clearing up homeless session... 2024-03-29T11:25:02.806 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.823+0000 7f31ef0c8640 10 client.4378.objecter successfully canceled tick 2024-03-29T11:25:02.806 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.823+0000 7f31ef0c8640 1 -- 172.21.15.114:0/611364186 >> v1:172.21.15.114:6816/4038755055 conn(0x7f31d406e5b0 legacy=0x7f31d4070a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:02.806 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.823+0000 7f31ef0c8640 10 monclient: shutdown 2024-03-29T11:25:02.806 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.823+0000 7f31ef0c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:02.806 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.823+0000 7f31ef0c8640 1 -- 172.21.15.114:0/611364186 >> v1:172.21.15.176:6789/0 conn(0x7f31e813ee90 legacy=0x7f31e81d6d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:02.810 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.827+0000 7f31ef0c8640 1 -- 172.21.15.114:0/611364186 shutdown_connections 2024-03-29T11:25:02.810 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.827+0000 7f31ef0c8640 1 -- 172.21.15.114:0/611364186 >> 172.21.15.114:0/611364186 conn(0x7f31e80b35e0 msgr2=0x7f31e813b120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:02.810 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.827+0000 7f31ef0c8640 1 -- 172.21.15.114:0/611364186 shutdown_connections 2024-03-29T11:25:02.811 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.827+0000 7f31ef0c8640 1 -- 172.21.15.114:0/611364186 wait complete. 2024-03-29T11:25:02.811 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:02.828+0000 7f31ef0c8640 1 librados: shutdown 2024-03-29T11:25:02.830 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:25:09.638 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:25:09.793 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.809+0000 7f34f359e640 10 monclient: get_monmap_and_config 2024-03-29T11:25:09.793 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.809+0000 7f34f359e640 10 monclient: build_initial_monmap 2024-03-29T11:25:09.793 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.809+0000 7f34f359e640 10 monclient: monmap: 2024-03-29T11:25:09.793 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:25:09.793 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:25:09.793 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:25:09.810892+0000 2024-03-29T11:25:09.794 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:25:09.810892+0000 2024-03-29T11:25:09.794 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:09.794 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:25:09.794 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:25:09.794 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T11:25:09.794 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T11:25:09.794 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:25:09.798 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.811+0000 7f34f359e640 1 Processor -- start 2024-03-29T11:25:09.798 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.811+0000 7f34f359e640 1 -- start start 2024-03-29T11:25:09.798 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.811+0000 7f34f359e640 10 monclient: init 2024-03-29T11:25:09.799 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.812+0000 7f34f359e640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:09.799 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.812+0000 7f34f359e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:25:09.799 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.812+0000 7f34f359e640 10 monclient(hunting): picked mon.noname-c con 0x7f34ec038c60 addr v1:172.21.15.176:6789/0 2024-03-29T11:25:09.799 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.812+0000 7f34f359e640 10 monclient(hunting): picked mon.noname-a con 0x7f34ec0324b0 addr v1:172.21.15.114:6789/0 2024-03-29T11:25:09.799 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.812+0000 7f34f359e640 10 monclient(hunting): picked mon.noname-b con 0x7f34ec0d11e0 addr v1:172.21.15.114:6790/0 2024-03-29T11:25:09.799 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.812+0000 7f34f359e640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34ec0caf80 con 0x7f34ec0324b0 2024-03-29T11:25:09.799 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.812+0000 7f34f359e640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34ec039850 con 0x7f34ec038c60 2024-03-29T11:25:09.799 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.812+0000 7f34f359e640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34ec0c78d0 con 0x7f34ec0d11e0 2024-03-29T11:25:09.799 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.812+0000 7f34f359e640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:09.799 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.812+0000 7f34f359e640 10 monclient(hunting): authenticate will time out at 1179.994273s 2024-03-29T11:25:09.802 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.817+0000 7f34f1d9b640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f34ec0324b0 0x7f34ec0cb4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:48979/0 (socket says 172.21.15.114:48979) 2024-03-29T11:25:09.802 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.817+0000 7f34f1d9b640 1 -- 172.21.15.114:0/86750958 learned_addr learned my addr 172.21.15.114:0/86750958 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:25:09.802 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.819+0000 7f34f159a640 1 -- 172.21.15.114:0/86750958 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1826012419 0 0) 0x7f34ec0caf80 con 0x7f34ec0324b0 2024-03-29T11:25:09.802 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.819+0000 7f34f159a640 10 monclient(hunting): _negotiate 2024-03-29T11:25:09.802 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.819+0000 7f34f159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:09.802 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.819+0000 7f34f159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:09.802 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.819+0000 7f34f159a640 10 monclient(hunting): my global_id is 4377 2024-03-29T11:25:09.803 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.819+0000 7f34f159a640 1 -- 172.21.15.114:0/86750958 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34dc003520 con 0x7f34ec0324b0 2024-03-29T11:25:09.803 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.819+0000 7f34f159a640 1 -- 172.21.15.114:0/86750958 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4040184527 0 0) 0x7f34ec039850 con 0x7f34ec038c60 2024-03-29T11:25:09.803 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.819+0000 7f34f159a640 10 monclient(hunting): _negotiate 2024-03-29T11:25:09.803 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.819+0000 7f34f159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:09.803 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.819+0000 7f34f159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:09.803 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.819+0000 7f34f159a640 10 monclient(hunting): my global_id is 4381 2024-03-29T11:25:09.803 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.819+0000 7f34f159a640 1 -- 172.21.15.114:0/86750958 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34ec0caf80 con 0x7f34ec038c60 2024-03-29T11:25:09.803 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.819+0000 7f34f159a640 1 -- 172.21.15.114:0/86750958 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1258878427 0 0) 0x7f34dc003520 con 0x7f34ec0324b0 2024-03-29T11:25:09.804 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.819+0000 7f34f159a640 1 -- 172.21.15.114:0/86750958 >> v1:172.21.15.114:6790/0 conn(0x7f34ec0d11e0 legacy=0x7f34ec0d35d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:25:09.804 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.819+0000 7f34f159a640 1 -- 172.21.15.114:0/86750958 >> v1:172.21.15.176:6789/0 conn(0x7f34ec038c60 legacy=0x7f34ec0c4ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:09.804 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.819+0000 7f34f159a640 10 monclient: _finish_hunting 0 2024-03-29T11:25:09.804 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.819+0000 7f34f159a640 1 monclient: found mon.noname-a 2024-03-29T11:25:09.804 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.819+0000 7f34f159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:09.804 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.819+0000 7f34f159a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.114:6789/0 2024-03-29T11:25:09.804 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.819+0000 7f34f159a640 1 -- 172.21.15.114:0/86750958 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34ec0c7a60 con 0x7f34ec0324b0 2024-03-29T11:25:09.804 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.819+0000 7f34f159a640 10 monclient: _finish_auth 0 2024-03-29T11:25:09.804 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.819+0000 7f34f159a640 10 monclient: _check_auth_tickets 2024-03-29T11:25:09.804 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.819+0000 7f34f159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:09.804 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.819+0000 7f34f159a640 1 -- 172.21.15.114:0/86750958 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f34e4001690 con 0x7f34ec0324b0 2024-03-29T11:25:09.804 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.819+0000 7f34f159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:09.804 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.819+0000 7f34f159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:25:09.804 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.819+0000 7f34f159a640 10 monclient: dump: 2024-03-29T11:25:09.805 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:25:09.805 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:25:09.805 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:25:09.805 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:25:09.805 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:09.805 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:25:09.805 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:25:09.805 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:25:09.805 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:25:09.805 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:25:09.805 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.819+0000 7f34f359e640 5 monclient: authenticate success, global_id 4377 2024-03-29T11:25:09.805 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.819+0000 7f34f359e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:09.806 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.821+0000 7f34f159a640 1 -- 172.21.15.114:0/86750958 <== mon.0 v1:172.21.15.114:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f34e4001860 con 0x7f34ec0324b0 2024-03-29T11:25:09.806 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.822+0000 7f34f159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:09.806 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.822+0000 7f34f159a640 1 -- 172.21.15.114:0/86750958 <== mon.0 v1:172.21.15.114:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f34e4001cb0 con 0x7f34ec0324b0 2024-03-29T11:25:09.806 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.822+0000 7f34f359e640 10 monclient: get_monmap_and_config success 2024-03-29T11:25:09.806 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.822+0000 7f34f359e640 4 set_mon_vals no callback set 2024-03-29T11:25:09.806 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.822+0000 7f34f159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:09.806 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.822+0000 7f34f159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:25:09.806 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.822+0000 7f34f159a640 10 monclient: dump: 2024-03-29T11:25:09.806 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:25:09.806 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:25:09.807 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:25:09.807 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:25:09.807 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:09.807 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:25:09.807 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:25:09.807 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:25:09.807 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:25:09.807 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:25:09.807 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.822+0000 7f34f359e640 10 monclient: shutdown 2024-03-29T11:25:09.807 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.822+0000 7f34f359e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:09.807 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.822+0000 7f34f359e640 1 -- 172.21.15.114:0/86750958 >> v1:172.21.15.114:6789/0 conn(0x7f34ec0324b0 legacy=0x7f34ec0cb4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:09.808 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.825+0000 7f34f359e640 1 -- 172.21.15.114:0/86750958 shutdown_connections 2024-03-29T11:25:09.808 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.825+0000 7f34f359e640 1 -- 172.21.15.114:0/86750958 >> 172.21.15.114:0/86750958 conn(0x7f34ec005c20 msgr2=0x7f34ec006030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:09.811 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.828+0000 7f34f359e640 1 -- 172.21.15.114:0/86750958 shutdown_connections 2024-03-29T11:25:09.811 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.828+0000 7f34f359e640 1 -- 172.21.15.114:0/86750958 wait complete. 2024-03-29T11:25:09.818 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.830+0000 7f34f359e640 10 monclient: build_initial_monmap 2024-03-29T11:25:09.818 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.830+0000 7f34f359e640 10 monclient: monmap: 2024-03-29T11:25:09.818 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:25:09.818 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:25:09.819 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T11:25:09.819 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T11:25:09.819 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:09.819 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:25:09.819 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:25:09.819 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T11:25:09.819 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T11:25:09.819 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:25:09.819 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.831+0000 7f34f359e640 1 librados: starting msgr at 2024-03-29T11:25:09.819 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.831+0000 7f34f359e640 1 librados: starting objecter 2024-03-29T11:25:09.819 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.831+0000 7f34f359e640 1 Processor -- start 2024-03-29T11:25:09.819 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.831+0000 7f34f359e640 1 -- start start 2024-03-29T11:25:09.819 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.831+0000 7f34f359e640 1 librados: setting wanted keys 2024-03-29T11:25:09.819 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.831+0000 7f34f359e640 1 librados: calling monclient init 2024-03-29T11:25:09.819 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.831+0000 7f34f359e640 10 monclient: init 2024-03-29T11:25:09.820 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.832+0000 7f34f359e640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:09.820 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.832+0000 7f34f359e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:25:09.820 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.832+0000 7f34f359e640 10 monclient(hunting): picked mon.noname-c con 0x7f34ec0324b0 addr v1:172.21.15.114:6790/0 2024-03-29T11:25:09.820 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.832+0000 7f34f359e640 10 monclient(hunting): picked mon.noname-a con 0x7f34ec038c60 addr v1:172.21.15.114:6789/0 2024-03-29T11:25:09.820 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.832+0000 7f34f359e640 10 monclient(hunting): picked mon.noname-b con 0x7f34ec0d11e0 addr v1:172.21.15.176:6789/0 2024-03-29T11:25:09.820 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.832+0000 7f34f359e640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34ec035310 con 0x7f34ec038c60 2024-03-29T11:25:09.820 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.832+0000 7f34f359e640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34ec0c4430 con 0x7f34ec0d11e0 2024-03-29T11:25:09.820 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.832+0000 7f34f359e640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34ec0c55b0 con 0x7f34ec0324b0 2024-03-29T11:25:09.820 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.832+0000 7f34f359e640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:09.820 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.832+0000 7f34f359e640 10 monclient(hunting): authenticate will time out at 1180.014222s 2024-03-29T11:25:09.825 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.840+0000 7f34f2d9d640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f34ec0d11e0 0x7f34ec174380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.114:38765/0 (socket says 172.21.15.114:38765) 2024-03-29T11:25:09.825 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.840+0000 7f34f2d9d640 1 -- 172.21.15.114:0/30632951 learned_addr learned my addr 172.21.15.114:0/30632951 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:25:09.825 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.842+0000 7f34e2ffd640 10 client.?.objecter ms_handle_connect 0x7f34ec0d11e0 2024-03-29T11:25:09.825 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.842+0000 7f34e2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:09.825 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.842+0000 7f34e2ffd640 10 client.?.objecter ms_handle_connect 0x7f34ec0324b0 2024-03-29T11:25:09.825 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.842+0000 7f34e2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:09.825 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.842+0000 7f34e2ffd640 10 client.?.objecter ms_handle_connect 0x7f34ec038c60 2024-03-29T11:25:09.826 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.842+0000 7f34e2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:09.826 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.842+0000 7f34e2ffd640 1 -- 172.21.15.114:0/30632951 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1595241122 0 0) 0x7f34ec035310 con 0x7f34ec038c60 2024-03-29T11:25:09.826 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.842+0000 7f34e2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:25:09.826 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.842+0000 7f34e2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:09.826 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.842+0000 7f34e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:09.827 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.842+0000 7f34e2ffd640 10 monclient(hunting): my global_id is 4380 2024-03-29T11:25:09.827 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.842+0000 7f34e2ffd640 1 -- 172.21.15.114:0/30632951 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34d4004eb0 con 0x7f34ec038c60 2024-03-29T11:25:09.827 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.842+0000 7f34e2ffd640 1 -- 172.21.15.114:0/30632951 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3481517038 0 0) 0x7f34ec0c55b0 con 0x7f34ec0324b0 2024-03-29T11:25:09.827 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.842+0000 7f34e2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:25:09.827 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.842+0000 7f34e2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:09.827 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.842+0000 7f34e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:09.827 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.842+0000 7f34e2ffd640 10 monclient(hunting): my global_id is 4394 2024-03-29T11:25:09.827 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.842+0000 7f34e2ffd640 1 -- 172.21.15.114:0/30632951 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34ec035310 con 0x7f34ec0324b0 2024-03-29T11:25:09.827 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.842+0000 7f34e2ffd640 1 -- 172.21.15.114:0/30632951 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1560455998 0 0) 0x7f34ec0c4430 con 0x7f34ec0d11e0 2024-03-29T11:25:09.827 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.843+0000 7f34e2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:25:09.827 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.843+0000 7f34e2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:09.827 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.843+0000 7f34e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:09.827 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.843+0000 7f34e2ffd640 10 monclient(hunting): my global_id is 4384 2024-03-29T11:25:09.828 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.843+0000 7f34e2ffd640 1 -- 172.21.15.114:0/30632951 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34ec0c55b0 con 0x7f34ec0d11e0 2024-03-29T11:25:09.828 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.843+0000 7f34e2ffd640 1 -- 172.21.15.114:0/30632951 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1619920882 0 0) 0x7f34ec035310 con 0x7f34ec0324b0 2024-03-29T11:25:09.828 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.843+0000 7f34e2ffd640 1 -- 172.21.15.114:0/30632951 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f34ec0c4430 con 0x7f34ec0324b0 2024-03-29T11:25:09.828 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.843+0000 7f34e2ffd640 1 -- 172.21.15.114:0/30632951 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f34f404fdd0 con 0x7f34ec0324b0 2024-03-29T11:25:09.828 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.843+0000 7f34e2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:09.828 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.843+0000 7f34e2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:25:09.828 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.843+0000 7f34e2ffd640 10 monclient(hunting): dump: 2024-03-29T11:25:09.828 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:25:09.828 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:25:09.828 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:25:09.828 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:25:09.828 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:09.828 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:25:09.828 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:25:09.828 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:25:09.829 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:25:09.829 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:25:09.829 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.843+0000 7f34e2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:25:09.829 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.843+0000 7f34e2ffd640 1 -- 172.21.15.114:0/30632951 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2497488561 0 0) 0x7f34ec0c55b0 con 0x7f34ec0d11e0 2024-03-29T11:25:09.829 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.843+0000 7f34e2ffd640 1 -- 172.21.15.114:0/30632951 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f34ec035310 con 0x7f34ec0d11e0 2024-03-29T11:25:09.829 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.843+0000 7f34e2ffd640 1 -- 172.21.15.114:0/30632951 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f34e8001710 con 0x7f34ec0d11e0 2024-03-29T11:25:09.829 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.843+0000 7f34e2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:09.829 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.843+0000 7f34e2ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:25:09.829 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.843+0000 7f34e2ffd640 10 monclient(hunting): dump: 2024-03-29T11:25:09.829 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:25:09.829 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:25:09.829 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:25:09.829 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:25:09.829 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:09.829 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:25:09.830 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:25:09.830 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:25:09.830 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:25:09.830 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:25:09.831 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.843+0000 7f34e2ffd640 1 -- 172.21.15.114:0/30632951 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4282440962 0 0) 0x7f34d4004eb0 con 0x7f34ec038c60 2024-03-29T11:25:09.831 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.843+0000 7f34e2ffd640 1 -- 172.21.15.114:0/30632951 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f34ec0c55b0 con 0x7f34ec038c60 2024-03-29T11:25:09.831 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.843+0000 7f34e2ffd640 1 -- 172.21.15.114:0/30632951 <== mon.2 v1:172.21.15.114:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1226001465 0 0) 0x7f34ec0c4430 con 0x7f34ec0324b0 2024-03-29T11:25:09.831 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.843+0000 7f34e2ffd640 1 -- 172.21.15.114:0/30632951 >> v1:172.21.15.176:6789/0 conn(0x7f34ec0d11e0 legacy=0x7f34ec174380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:09.831 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.843+0000 7f34e2ffd640 1 -- 172.21.15.114:0/30632951 >> v1:172.21.15.114:6789/0 conn(0x7f34ec038c60 legacy=0x7f34ec0c4ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:09.832 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.843+0000 7f34e2ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:25:09.832 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.843+0000 7f34e2ffd640 1 monclient: found mon.c 2024-03-29T11:25:09.832 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.843+0000 7f34e2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:09.832 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.843+0000 7f34e2ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:25:09.832 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.843+0000 7f34e2ffd640 1 -- 172.21.15.114:0/30632951 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34ec0355f0 con 0x7f34ec0324b0 2024-03-29T11:25:09.832 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.843+0000 7f34e2ffd640 10 monclient: _finish_auth 0 2024-03-29T11:25:09.832 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.843+0000 7f34e2ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:25:09.832 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.843+0000 7f34e2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:09.832 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.846+0000 7f34f359e640 5 monclient: authenticate success, global_id 4394 2024-03-29T11:25:09.832 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.846+0000 7f34f359e640 10 monclient: _renew_subs 2024-03-29T11:25:09.832 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.846+0000 7f34f359e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:25:09.832 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.846+0000 7f34f359e640 1 -- 172.21.15.114:0/30632951 --> v1:172.21.15.114:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34ec0343d0 con 0x7f34ec0324b0 2024-03-29T11:25:09.832 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.846+0000 7f34f359e640 10 client.4394.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:25:09.832 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.846+0000 7f34f359e640 10 monclient: _renew_subs 2024-03-29T11:25:09.833 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.846+0000 7f34f359e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:25:09.833 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.846+0000 7f34f359e640 1 -- 172.21.15.114:0/30632951 --> v1:172.21.15.114:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f34ec037420 con 0x7f34ec0324b0 2024-03-29T11:25:09.833 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.847+0000 7f34f359e640 1 librados: init done 2024-03-29T11:25:09.833 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.848+0000 7f34e2ffd640 1 -- 172.21.15.114:0/30632951 <== mon.2 v1:172.21.15.114:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f34f404f400 con 0x7f34ec0324b0 2024-03-29T11:25:09.833 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.848+0000 7f34e2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:09.833 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.848+0000 7f34e2ffd640 1 -- 172.21.15.114:0/30632951 <== mon.2 v1:172.21.15.114:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f34f404f730 con 0x7f34ec0324b0 2024-03-29T11:25:09.833 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.848+0000 7f34e2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:09.833 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.848+0000 7f34e2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:25:09.833 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.848+0000 7f34e2ffd640 10 monclient: dump: 2024-03-29T11:25:09.833 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:25:09.833 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:25:09.833 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:25:09.833 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:25:09.833 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:09.833 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:25:09.834 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:25:09.834 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:25:09.834 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:25:09.834 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:25:09.834 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.848+0000 7f34f359e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:09.834 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.848+0000 7f34f359e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:09.834 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.848+0000 7f34f359e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:25:09.834 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.848+0000 7f34f359e640 1 -- 172.21.15.114:0/30632951 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34ec007b90 con 0x7f34ec0324b0 2024-03-29T11:25:09.834 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.849+0000 7f34f0d99640 4 set_mon_vals no callback set 2024-03-29T11:25:09.835 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.852+0000 7f34e2ffd640 1 -- 172.21.15.114:0/30632951 <== mon.2 v1:172.21.15.114:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1594292461 0 0) 0x7f34f4067bf0 con 0x7f34ec0324b0 2024-03-29T11:25:09.836 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.852+0000 7f34e2ffd640 1 -- 172.21.15.114:0/30632951 <== mon.2 v1:172.21.15.114:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3238987313 0 0) 0x7f34f40fdff0 con 0x7f34ec0324b0 2024-03-29T11:25:09.836 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.852+0000 7f34e2ffd640 10 client.4394.objecter ms_dispatch 0x7f34f4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:25:09.836 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.852+0000 7f34e2ffd640 3 client.4394.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:25:09.836 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.852+0000 7f34e2ffd640 3 client.4394.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:25:09.837 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.853+0000 7f34e2ffd640 20 client.4394.objecter dump_active .. 0 homeless 2024-03-29T11:25:09.839 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.856+0000 7f34f1d9b640 10 monclient: get_auth_request con 0x7f34d406dea0 auth_method 0 2024-03-29T11:25:09.840 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.857+0000 7f34e2ffd640 10 client.4394.objecter ms_handle_connect 0x7f34d406dea0 2024-03-29T11:25:09.845 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.860+0000 7f34e2ffd640 1 -- 172.21.15.114:0/30632951 <== mon.2 v1:172.21.15.114:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f34f40c98d0 con 0x7f34ec0324b0 2024-03-29T11:25:09.845 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.860+0000 7f34e2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:09.845 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:09.860+0000 7f34e2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:25:10.164 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:10.180+0000 7f34f359e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:10.164 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:10.180+0000 7f34f359e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:10.164 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:10.180+0000 7f34f359e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:25:10.164 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:10.180+0000 7f34f359e640 1 -- 172.21.15.114:0/30632951 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f34ec004cd0 con 0x7f34ec0324b0 2024-03-29T11:25:10.164 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:10.181+0000 7f34e2ffd640 1 -- 172.21.15.114:0/30632951 <== mon.2 v1:172.21.15.114:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2858224613) 0x7f34f40cb800 con 0x7f34ec0324b0 2024-03-29T11:25:10.165 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:10.181+0000 7f34e2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:10.165 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:10.181+0000 7f34e2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:25:10.165 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T11:25:10.165 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"b0de2b9b-c45a-44c7-b6ee-e83b39a1e108","created":"2024-03-29T11:20:49.420660+0000","modified":"2024-03-29T11:20:51.700167+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:51.700167+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a64785d3-6b11-499a-b706-cc827ab96a74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5882dadb-e7a8-4604-ba9a-418013c106c8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d7e8c409-bc66-4e4a-bbdb-420bad31388e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0682989-3595-4cd6-90a5-ab9111c278e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"67068120-e3ff-4817-bd06-44feb7479a35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09e8b-8a85-4997-bdea-75d754e6ee67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2f0d9ead-f47f-4cb3-92a4-0a5ad23cd831","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fdb6a25f-6bd8-4c65-9b9b-b460ba7673a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:25:10.168 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:10.184+0000 7f34f359e640 10 librados: watch_flush enter 2024-03-29T11:25:10.168 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:10.185+0000 7f34f359e640 10 librados: watch_flush exit 2024-03-29T11:25:10.168 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:10.185+0000 7f34f359e640 20 client.4394.objecter shutdown clearing up homeless session... 2024-03-29T11:25:10.168 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:10.185+0000 7f34f359e640 10 client.4394.objecter successfully canceled tick 2024-03-29T11:25:10.168 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:10.185+0000 7f34f359e640 1 -- 172.21.15.114:0/30632951 >> v1:172.21.15.114:6816/4038755055 conn(0x7f34d406dea0 legacy=0x7f34d4070360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:10.168 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:10.185+0000 7f34f359e640 10 monclient: shutdown 2024-03-29T11:25:10.168 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:10.185+0000 7f34f359e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:10.168 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:10.185+0000 7f34f359e640 1 -- 172.21.15.114:0/30632951 >> v1:172.21.15.114:6790/0 conn(0x7f34ec0324b0 legacy=0x7f34ec0c6c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:10.172 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:10.188+0000 7f34f359e640 1 -- 172.21.15.114:0/30632951 shutdown_connections 2024-03-29T11:25:10.172 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:10.188+0000 7f34f359e640 1 -- 172.21.15.114:0/30632951 >> 172.21.15.114:0/30632951 conn(0x7f34ec005c20 msgr2=0x7f34ec03ac30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:10.172 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:10.189+0000 7f34f359e640 1 -- 172.21.15.114:0/30632951 shutdown_connections 2024-03-29T11:25:10.173 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:10.189+0000 7f34f359e640 1 -- 172.21.15.114:0/30632951 wait complete. 2024-03-29T11:25:10.173 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:10.189+0000 7f34f359e640 1 librados: shutdown 2024-03-29T11:25:10.192 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:25:16.998 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:25:17.154 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.170+0000 7f04e6704640 10 monclient: get_monmap_and_config 2024-03-29T11:25:17.154 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.170+0000 7f04e6704640 10 monclient: build_initial_monmap 2024-03-29T11:25:17.154 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.170+0000 7f04e6704640 10 monclient: monmap: 2024-03-29T11:25:17.154 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:25:17.154 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:25:17.155 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:25:17.171804+0000 2024-03-29T11:25:17.155 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:25:17.171804+0000 2024-03-29T11:25:17.155 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:17.155 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:25:17.155 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:25:17.155 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T11:25:17.155 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T11:25:17.155 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:25:17.157 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.171+0000 7f04e6704640 1 Processor -- start 2024-03-29T11:25:17.157 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.171+0000 7f04e6704640 1 -- start start 2024-03-29T11:25:17.157 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.172+0000 7f04e6704640 10 monclient: init 2024-03-29T11:25:17.157 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.173+0000 7f04e6704640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:17.157 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.173+0000 7f04e6704640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:25:17.157 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.173+0000 7f04e6704640 10 monclient(hunting): picked mon.noname-b con 0x7f04e00b0860 addr v1:172.21.15.114:6790/0 2024-03-29T11:25:17.157 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.173+0000 7f04e6704640 10 monclient(hunting): picked mon.noname-a con 0x7f04e013ef30 addr v1:172.21.15.114:6789/0 2024-03-29T11:25:17.158 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.173+0000 7f04e6704640 10 monclient(hunting): picked mon.noname-c con 0x7f04e00af8e0 addr v1:172.21.15.176:6789/0 2024-03-29T11:25:17.158 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.173+0000 7f04e6704640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04e00b0c40 con 0x7f04e013ef30 2024-03-29T11:25:17.158 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.173+0000 7f04e6704640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04e00aed10 con 0x7f04e00af8e0 2024-03-29T11:25:17.158 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.173+0000 7f04e6704640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04e00afd50 con 0x7f04e00b0860 2024-03-29T11:25:17.158 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.173+0000 7f04e6704640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:17.158 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.173+0000 7f04dffff640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f04e00b0860 0x7f04e013e820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:34326/0 (socket says 172.21.15.114:34326) 2024-03-29T11:25:17.158 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.173+0000 7f04e4ca1640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f04e00af8e0 0x7f04e0149a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.114:33036/0 (socket says 172.21.15.114:33036) 2024-03-29T11:25:17.158 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.173+0000 7f04dffff640 1 -- 172.21.15.114:0/1106947464 learned_addr learned my addr 172.21.15.114:0/1106947464 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:25:17.158 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.173+0000 7f04e6704640 10 monclient(hunting): authenticate will time out at 1187.355626s 2024-03-29T11:25:17.161 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.174+0000 7f04deffd640 1 -- 172.21.15.114:0/1106947464 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3558215613 0 0) 0x7f04e00afd50 con 0x7f04e00b0860 2024-03-29T11:25:17.161 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.174+0000 7f04deffd640 10 monclient(hunting): _negotiate 2024-03-29T11:25:17.161 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.174+0000 7f04deffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:17.161 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.174+0000 7f04deffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:17.161 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.174+0000 7f04deffd640 10 monclient(hunting): my global_id is 4397 2024-03-29T11:25:17.161 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.174+0000 7f04deffd640 1 -- 172.21.15.114:0/1106947464 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04c8003520 con 0x7f04e00b0860 2024-03-29T11:25:17.161 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.174+0000 7f04deffd640 1 -- 172.21.15.114:0/1106947464 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4287631703 0 0) 0x7f04e00aed10 con 0x7f04e00af8e0 2024-03-29T11:25:17.161 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.174+0000 7f04deffd640 10 monclient(hunting): _negotiate 2024-03-29T11:25:17.161 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.174+0000 7f04deffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:17.161 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.174+0000 7f04deffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:17.161 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.174+0000 7f04deffd640 10 monclient(hunting): my global_id is 4387 2024-03-29T11:25:17.161 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.174+0000 7f04deffd640 1 -- 172.21.15.114:0/1106947464 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04e00afd50 con 0x7f04e00af8e0 2024-03-29T11:25:17.161 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.174+0000 7f04deffd640 1 -- 172.21.15.114:0/1106947464 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2351315314 0 0) 0x7f04c8003520 con 0x7f04e00b0860 2024-03-29T11:25:17.162 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.174+0000 7f04deffd640 1 -- 172.21.15.114:0/1106947464 >> v1:172.21.15.176:6789/0 conn(0x7f04e00af8e0 legacy=0x7f04e0149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:17.162 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.175+0000 7f04deffd640 1 -- 172.21.15.114:0/1106947464 >> v1:172.21.15.114:6789/0 conn(0x7f04e013ef30 legacy=0x7f04e00af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:17.162 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.175+0000 7f04deffd640 10 monclient: _finish_hunting 0 2024-03-29T11:25:17.162 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.175+0000 7f04deffd640 1 monclient: found mon.noname-b 2024-03-29T11:25:17.162 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.175+0000 7f04deffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:17.162 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.175+0000 7f04deffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.114:6790/0 2024-03-29T11:25:17.162 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.175+0000 7f04deffd640 1 -- 172.21.15.114:0/1106947464 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04e00ab1b0 con 0x7f04e00b0860 2024-03-29T11:25:17.162 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.175+0000 7f04deffd640 10 monclient: _finish_auth 0 2024-03-29T11:25:17.162 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.175+0000 7f04deffd640 10 monclient: _check_auth_tickets 2024-03-29T11:25:17.162 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.175+0000 7f04deffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:17.162 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.175+0000 7f04deffd640 1 -- 172.21.15.114:0/1106947464 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f04d0001690 con 0x7f04e00b0860 2024-03-29T11:25:17.162 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.175+0000 7f04deffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:17.162 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.175+0000 7f04deffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:25:17.163 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.175+0000 7f04deffd640 10 monclient: dump: 2024-03-29T11:25:17.163 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:25:17.163 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:25:17.163 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:25:17.163 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:25:17.163 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:17.163 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:25:17.163 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:25:17.163 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:25:17.163 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:25:17.163 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:25:17.163 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.175+0000 7f04e6704640 5 monclient: authenticate success, global_id 4397 2024-03-29T11:25:17.163 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.175+0000 7f04e6704640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:17.163 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.175+0000 7f04deffd640 1 -- 172.21.15.114:0/1106947464 <== mon.2 v1:172.21.15.114:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f04d00017f0 con 0x7f04e00b0860 2024-03-29T11:25:17.164 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.175+0000 7f04deffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:17.164 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.175+0000 7f04deffd640 1 -- 172.21.15.114:0/1106947464 <== mon.2 v1:172.21.15.114:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f04d0001c40 con 0x7f04e00b0860 2024-03-29T11:25:17.164 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.175+0000 7f04deffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:17.164 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.175+0000 7f04deffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:25:17.164 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.175+0000 7f04deffd640 10 monclient: dump: 2024-03-29T11:25:17.164 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:25:17.164 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:25:17.164 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:25:17.164 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:25:17.164 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:17.164 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:25:17.164 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:25:17.164 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:25:17.165 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:25:17.165 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:25:17.165 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.175+0000 7f04e6704640 10 monclient: get_monmap_and_config success 2024-03-29T11:25:17.165 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.175+0000 7f04e6704640 4 set_mon_vals no callback set 2024-03-29T11:25:17.165 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.176+0000 7f04e6704640 10 monclient: shutdown 2024-03-29T11:25:17.165 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.176+0000 7f04e6704640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:17.165 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.176+0000 7f04e6704640 1 -- 172.21.15.114:0/1106947464 >> v1:172.21.15.114:6790/0 conn(0x7f04e00b0860 legacy=0x7f04e013e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:17.165 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.177+0000 7f04e6704640 1 -- 172.21.15.114:0/1106947464 shutdown_connections 2024-03-29T11:25:17.165 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.177+0000 7f04e6704640 1 -- 172.21.15.114:0/1106947464 >> 172.21.15.114:0/1106947464 conn(0x7f04e006b660 msgr2=0x7f04e006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:17.165 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.177+0000 7f04e6704640 1 -- 172.21.15.114:0/1106947464 shutdown_connections 2024-03-29T11:25:17.165 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.177+0000 7f04e6704640 1 -- 172.21.15.114:0/1106947464 wait complete. 2024-03-29T11:25:17.169 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.178+0000 7f04e6704640 10 monclient: build_initial_monmap 2024-03-29T11:25:17.170 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.178+0000 7f04e6704640 10 monclient: monmap: 2024-03-29T11:25:17.170 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:25:17.170 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:25:17.170 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T11:25:17.170 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T11:25:17.170 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:17.170 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:25:17.170 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:25:17.170 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T11:25:17.170 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T11:25:17.170 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:25:17.170 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.178+0000 7f04e6704640 1 librados: starting msgr at 2024-03-29T11:25:17.171 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.178+0000 7f04e6704640 1 librados: starting objecter 2024-03-29T11:25:17.171 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.179+0000 7f04e6704640 1 Processor -- start 2024-03-29T11:25:17.171 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.179+0000 7f04e6704640 1 -- start start 2024-03-29T11:25:17.171 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.179+0000 7f04e6704640 1 librados: setting wanted keys 2024-03-29T11:25:17.171 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.179+0000 7f04e6704640 1 librados: calling monclient init 2024-03-29T11:25:17.171 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.179+0000 7f04e6704640 10 monclient: init 2024-03-29T11:25:17.171 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.180+0000 7f04e6704640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:17.171 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.180+0000 7f04e6704640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:25:17.171 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.180+0000 7f04e6704640 10 monclient(hunting): picked mon.noname-c con 0x7f04e00af8e0 addr v1:172.21.15.114:6790/0 2024-03-29T11:25:17.171 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.180+0000 7f04e6704640 10 monclient(hunting): picked mon.noname-b con 0x7f04e00b0860 addr v1:172.21.15.176:6789/0 2024-03-29T11:25:17.171 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.180+0000 7f04e6704640 10 monclient(hunting): picked mon.noname-a con 0x7f04e013ef30 addr v1:172.21.15.114:6789/0 2024-03-29T11:25:17.171 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.180+0000 7f04e6704640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04e00b4180 con 0x7f04e013ef30 2024-03-29T11:25:17.171 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.180+0000 7f04e6704640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04e01d6770 con 0x7f04e00b0860 2024-03-29T11:25:17.172 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.180+0000 7f04e6704640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04e01d7310 con 0x7f04e00af8e0 2024-03-29T11:25:17.172 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.180+0000 7f04e6704640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:17.172 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.180+0000 7f04e6704640 10 monclient(hunting): authenticate will time out at 1187.362799s 2024-03-29T11:25:17.172 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.180+0000 7f04e4ca1640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f04e013ef30 0x7f04e01ddb30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:55178/0 (socket says 172.21.15.114:55178) 2024-03-29T11:25:17.172 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.180+0000 7f04e4ca1640 1 -- 172.21.15.114:0/4079054332 learned_addr learned my addr 172.21.15.114:0/4079054332 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:25:17.172 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.180+0000 7f04dcff9640 10 client.?.objecter ms_handle_connect 0x7f04e013ef30 2024-03-29T11:25:17.172 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.181+0000 7f04dcff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:17.172 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.181+0000 7f04dcff9640 10 client.?.objecter ms_handle_connect 0x7f04e00b0860 2024-03-29T11:25:17.172 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.181+0000 7f04dcff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:17.172 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.181+0000 7f04dcff9640 10 client.?.objecter ms_handle_connect 0x7f04e00af8e0 2024-03-29T11:25:17.172 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.181+0000 7f04dcff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:17.172 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.181+0000 7f04dcff9640 1 -- 172.21.15.114:0/4079054332 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3049311791 0 0) 0x7f04e01d6770 con 0x7f04e00b0860 2024-03-29T11:25:17.172 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.181+0000 7f04dcff9640 10 monclient(hunting): _negotiate 2024-03-29T11:25:17.173 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.181+0000 7f04dcff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:17.173 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.181+0000 7f04dcff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:17.173 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.181+0000 7f04dcff9640 10 monclient(hunting): my global_id is 4390 2024-03-29T11:25:17.173 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.181+0000 7f04dcff9640 1 -- 172.21.15.114:0/4079054332 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04cc004eb0 con 0x7f04e00b0860 2024-03-29T11:25:17.173 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.182+0000 7f04dcff9640 1 -- 172.21.15.114:0/4079054332 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2849477480 0 0) 0x7f04cc004eb0 con 0x7f04e00b0860 2024-03-29T11:25:17.173 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.182+0000 7f04dcff9640 1 -- 172.21.15.114:0/4079054332 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f04e01d6770 con 0x7f04e00b0860 2024-03-29T11:25:17.173 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.182+0000 7f04dcff9640 1 -- 172.21.15.114:0/4079054332 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f04d4001810 con 0x7f04e00b0860 2024-03-29T11:25:17.173 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.182+0000 7f04dcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:17.173 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.182+0000 7f04dcff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:25:17.173 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.182+0000 7f04dcff9640 10 monclient(hunting): dump: 2024-03-29T11:25:17.173 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:25:17.173 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:25:17.173 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:25:17.173 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:25:17.174 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:17.174 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:25:17.174 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:25:17.174 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:25:17.174 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:25:17.174 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:25:17.174 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.182+0000 7f04dcff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:25:17.174 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.182+0000 7f04dcff9640 1 -- 172.21.15.114:0/4079054332 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 173431190 0 0) 0x7f04e01d7310 con 0x7f04e00af8e0 2024-03-29T11:25:17.174 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.182+0000 7f04dcff9640 10 monclient(hunting): _negotiate 2024-03-29T11:25:17.174 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.182+0000 7f04dcff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:17.174 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.182+0000 7f04dcff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:17.174 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.182+0000 7f04dcff9640 10 monclient(hunting): my global_id is 4400 2024-03-29T11:25:17.174 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.182+0000 7f04dcff9640 1 -- 172.21.15.114:0/4079054332 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04cc004eb0 con 0x7f04e00af8e0 2024-03-29T11:25:17.175 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.182+0000 7f04dcff9640 1 -- 172.21.15.114:0/4079054332 <== mon.1 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3122211984 0 0) 0x7f04e01d6770 con 0x7f04e00b0860 2024-03-29T11:25:17.175 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.182+0000 7f04dcff9640 1 -- 172.21.15.114:0/4079054332 >> v1:172.21.15.114:6790/0 conn(0x7f04e00af8e0 legacy=0x7f04e01d6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:17.175 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.182+0000 7f04dcff9640 1 -- 172.21.15.114:0/4079054332 >> v1:172.21.15.114:6789/0 conn(0x7f04e013ef30 legacy=0x7f04e01ddb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:17.175 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.182+0000 7f04dcff9640 10 monclient: _finish_hunting 0 2024-03-29T11:25:17.175 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.182+0000 7f04dcff9640 1 monclient: found mon.b 2024-03-29T11:25:17.175 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.182+0000 7f04dcff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:17.175 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.182+0000 7f04dcff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:25:17.175 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.182+0000 7f04dcff9640 1 -- 172.21.15.114:0/4079054332 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04e01d74a0 con 0x7f04e00b0860 2024-03-29T11:25:17.175 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.182+0000 7f04dcff9640 10 monclient: _finish_auth 0 2024-03-29T11:25:17.175 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.182+0000 7f04dcff9640 10 monclient: _check_auth_tickets 2024-03-29T11:25:17.175 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.182+0000 7f04dcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:17.175 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.182+0000 7f04e6704640 5 monclient: authenticate success, global_id 4390 2024-03-29T11:25:17.175 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.182+0000 7f04e6704640 10 monclient: _renew_subs 2024-03-29T11:25:17.176 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.182+0000 7f04e6704640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:25:17.176 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.182+0000 7f04e6704640 1 -- 172.21.15.114:0/4079054332 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04e00b1a70 con 0x7f04e00b0860 2024-03-29T11:25:17.176 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.182+0000 7f04e6704640 10 client.4390.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:25:17.176 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.182+0000 7f04e6704640 10 monclient: _renew_subs 2024-03-29T11:25:17.176 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.182+0000 7f04e6704640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:25:17.176 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.182+0000 7f04e6704640 1 -- 172.21.15.114:0/4079054332 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f04e00b2f50 con 0x7f04e00b0860 2024-03-29T11:25:17.176 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.182+0000 7f04e6704640 1 librados: init done 2024-03-29T11:25:17.176 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.183+0000 7f04dcff9640 1 -- 172.21.15.114:0/4079054332 <== mon.1 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f04d4001d40 con 0x7f04e00b0860 2024-03-29T11:25:17.176 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.183+0000 7f04dcff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:17.176 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.183+0000 7f04de7fc640 4 set_mon_vals no callback set 2024-03-29T11:25:17.176 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.183+0000 7f04dcff9640 1 -- 172.21.15.114:0/4079054332 <== mon.1 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f04d40037b0 con 0x7f04e00b0860 2024-03-29T11:25:17.176 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.183+0000 7f04dcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:17.176 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.183+0000 7f04dcff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:25:17.177 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.183+0000 7f04dcff9640 10 monclient: dump: 2024-03-29T11:25:17.177 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:25:17.177 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:25:17.177 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:25:17.177 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:25:17.177 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:17.177 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:25:17.177 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:25:17.177 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:25:17.177 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:25:17.177 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:25:17.177 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.184+0000 7f04e6704640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:17.177 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.184+0000 7f04e6704640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:17.178 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.184+0000 7f04e6704640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:25:17.178 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.185+0000 7f04e6704640 1 -- 172.21.15.114:0/4079054332 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04c0005080 con 0x7f04e00b0860 2024-03-29T11:25:17.178 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.185+0000 7f04dcff9640 1 -- 172.21.15.114:0/4079054332 <== mon.1 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1594292461 0 0) 0x7f04d4003910 con 0x7f04e00b0860 2024-03-29T11:25:17.178 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.186+0000 7f04dcff9640 1 -- 172.21.15.114:0/4079054332 <== mon.1 v1:172.21.15.176:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3238987313 0 0) 0x7f04d40837e0 con 0x7f04e00b0860 2024-03-29T11:25:17.178 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.186+0000 7f04dcff9640 10 client.4390.objecter ms_dispatch 0x7f04e0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:25:17.178 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.186+0000 7f04dcff9640 3 client.4390.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:25:17.178 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.186+0000 7f04dcff9640 3 client.4390.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:25:17.178 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.186+0000 7f04dcff9640 20 client.4390.objecter dump_active .. 0 homeless 2024-03-29T11:25:17.178 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.186+0000 7f04dffff640 10 monclient: get_auth_request con 0x7f04cc06dd30 auth_method 0 2024-03-29T11:25:17.179 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.186+0000 7f04dcff9640 10 client.4390.objecter ms_handle_connect 0x7f04cc06dd30 2024-03-29T11:25:17.179 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.192+0000 7f04dcff9640 1 -- 172.21.15.114:0/4079054332 <== mon.1 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f04d404eef0 con 0x7f04e00b0860 2024-03-29T11:25:17.179 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.192+0000 7f04dcff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:17.179 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.192+0000 7f04dcff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:25:17.587 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.603+0000 7f04e6704640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:17.587 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.603+0000 7f04e6704640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:17.587 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.603+0000 7f04e6704640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:25:17.587 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.603+0000 7f04e6704640 1 -- 172.21.15.114:0/4079054332 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f04c0005330 con 0x7f04e00b0860 2024-03-29T11:25:17.588 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.604+0000 7f04dcff9640 1 -- 172.21.15.114:0/4079054332 <== mon.1 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2858224613) 0x7f04d4050ff0 con 0x7f04e00b0860 2024-03-29T11:25:17.588 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T11:25:17.588 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"b0de2b9b-c45a-44c7-b6ee-e83b39a1e108","created":"2024-03-29T11:20:49.420660+0000","modified":"2024-03-29T11:20:51.700167+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:51.700167+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a64785d3-6b11-499a-b706-cc827ab96a74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5882dadb-e7a8-4604-ba9a-418013c106c8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d7e8c409-bc66-4e4a-bbdb-420bad31388e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0682989-3595-4cd6-90a5-ab9111c278e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"67068120-e3ff-4817-bd06-44feb7479a35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09e8b-8a85-4997-bdea-75d754e6ee67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2f0d9ead-f47f-4cb3-92a4-0a5ad23cd831","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fdb6a25f-6bd8-4c65-9b9b-b460ba7673a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:25:17.589 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.604+0000 7f04dcff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:17.589 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.604+0000 7f04dcff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:25:17.594 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.608+0000 7f04e6704640 10 librados: watch_flush enter 2024-03-29T11:25:17.594 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.608+0000 7f04e6704640 10 librados: watch_flush exit 2024-03-29T11:25:17.594 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.608+0000 7f04e6704640 20 client.4390.objecter shutdown clearing up homeless session... 2024-03-29T11:25:17.594 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.608+0000 7f04e6704640 10 client.4390.objecter successfully canceled tick 2024-03-29T11:25:17.594 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.609+0000 7f04e6704640 1 -- 172.21.15.114:0/4079054332 >> v1:172.21.15.114:6816/4038755055 conn(0x7f04cc06dd30 legacy=0x7f04cc0701f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:17.595 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.609+0000 7f04e6704640 10 monclient: shutdown 2024-03-29T11:25:17.595 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.609+0000 7f04e6704640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:17.595 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.609+0000 7f04e6704640 1 -- 172.21.15.114:0/4079054332 >> v1:172.21.15.176:6789/0 conn(0x7f04e00b0860 legacy=0x7f04e00b2660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:17.595 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.609+0000 7f04e6704640 1 -- 172.21.15.114:0/4079054332 shutdown_connections 2024-03-29T11:25:17.595 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.609+0000 7f04e6704640 1 -- 172.21.15.114:0/4079054332 >> 172.21.15.114:0/4079054332 conn(0x7f04e006b660 msgr2=0x7f04e013e5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:17.595 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.609+0000 7f04e6704640 1 -- 172.21.15.114:0/4079054332 shutdown_connections 2024-03-29T11:25:17.595 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.609+0000 7f04e6704640 1 -- 172.21.15.114:0/4079054332 wait complete. 2024-03-29T11:25:17.596 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:17.609+0000 7f04e6704640 1 librados: shutdown 2024-03-29T11:25:17.607 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:25:24.415 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:25:24.557 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.573+0000 7f4fae139640 10 monclient: get_monmap_and_config 2024-03-29T11:25:24.558 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.573+0000 7f4fae139640 10 monclient: build_initial_monmap 2024-03-29T11:25:24.558 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.573+0000 7f4fae139640 10 monclient: monmap: 2024-03-29T11:25:24.558 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:25:24.558 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:25:24.558 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:25:24.574968+0000 2024-03-29T11:25:24.558 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:25:24.574968+0000 2024-03-29T11:25:24.558 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:24.558 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:25:24.558 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:25:24.558 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T11:25:24.558 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T11:25:24.558 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:25:24.559 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.575+0000 7f4fae139640 1 Processor -- start 2024-03-29T11:25:24.559 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.575+0000 7f4fae139640 1 -- start start 2024-03-29T11:25:24.559 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.575+0000 7f4fae139640 10 monclient: init 2024-03-29T11:25:24.560 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.576+0000 7f4fae139640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:24.560 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.576+0000 7f4fae139640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:25:24.560 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.576+0000 7f4fae139640 10 monclient(hunting): picked mon.noname-a con 0x7f4fa8070170 addr v1:172.21.15.114:6789/0 2024-03-29T11:25:24.560 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.576+0000 7f4fae139640 10 monclient(hunting): picked mon.noname-b con 0x7f4fa80713f0 addr v1:172.21.15.114:6790/0 2024-03-29T11:25:24.560 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.576+0000 7f4fae139640 10 monclient(hunting): picked mon.noname-c con 0x7f4fa8070550 addr v1:172.21.15.176:6789/0 2024-03-29T11:25:24.560 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.576+0000 7f4fae139640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fa813dbd0 con 0x7f4fa8070170 2024-03-29T11:25:24.560 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.576+0000 7f4fae139640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fa813ff70 con 0x7f4fa8070550 2024-03-29T11:25:24.560 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.576+0000 7f4fa6ffd640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f4fa80713f0 0x7f4fa81406e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:39434/0 (socket says 172.21.15.114:39434) 2024-03-29T11:25:24.560 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.576+0000 7f4fa6ffd640 1 -- 172.21.15.114:0/2350089989 learned_addr learned my addr 172.21.15.114:0/2350089989 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:25:24.560 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.576+0000 7f4fae139640 1 -- 172.21.15.114:0/2350089989 --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fa8140250 con 0x7f4fa80713f0 2024-03-29T11:25:24.560 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.576+0000 7f4fae139640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:24.560 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.576+0000 7f4fae139640 10 monclient(hunting): authenticate will time out at 1194.758630s 2024-03-29T11:25:24.561 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.577+0000 7f4fa67fc640 1 -- 172.21.15.114:0/2350089989 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2734197744 0 0) 0x7f4fa813ff70 con 0x7f4fa8070550 2024-03-29T11:25:24.561 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.577+0000 7f4fa67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:25:24.561 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.577+0000 7f4fa67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:24.561 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.577+0000 7f4fa67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:24.561 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.577+0000 7f4fa67fc640 10 monclient(hunting): my global_id is 4393 2024-03-29T11:25:24.561 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.577+0000 7f4fa67fc640 1 -- 172.21.15.114:0/2350089989 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f90003520 con 0x7f4fa8070550 2024-03-29T11:25:24.561 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.577+0000 7f4fa67fc640 1 -- 172.21.15.114:0/2350089989 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 127345781 0 0) 0x7f4fa8140250 con 0x7f4fa80713f0 2024-03-29T11:25:24.561 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.577+0000 7f4fa67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:25:24.562 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.577+0000 7f4fa67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:24.562 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.577+0000 7f4fa67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:24.562 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.577+0000 7f4fa67fc640 10 monclient(hunting): my global_id is 4403 2024-03-29T11:25:24.562 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.577+0000 7f4fa67fc640 1 -- 172.21.15.114:0/2350089989 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4fa813ff70 con 0x7f4fa80713f0 2024-03-29T11:25:24.562 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.577+0000 7f4fa67fc640 1 -- 172.21.15.114:0/2350089989 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1344970702 0 0) 0x7f4f90003520 con 0x7f4fa8070550 2024-03-29T11:25:24.562 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.577+0000 7f4fa67fc640 1 -- 172.21.15.114:0/2350089989 >> v1:172.21.15.114:6790/0 conn(0x7f4fa80713f0 legacy=0x7f4fa81406e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:24.562 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.577+0000 7f4fa67fc640 1 -- 172.21.15.114:0/2350089989 >> v1:172.21.15.114:6789/0 conn(0x7f4fa8070170 legacy=0x7f4fa813d4c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:25:24.565 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.580+0000 7f4fa67fc640 10 monclient: _finish_hunting 0 2024-03-29T11:25:24.565 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.580+0000 7f4fa67fc640 1 monclient: found mon.noname-c 2024-03-29T11:25:24.565 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.580+0000 7f4fa67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:24.565 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.580+0000 7f4fa67fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.176:6789/0 2024-03-29T11:25:24.565 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.580+0000 7f4fa67fc640 1 -- 172.21.15.114:0/2350089989 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fa8140df0 con 0x7f4fa8070550 2024-03-29T11:25:24.566 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.580+0000 7f4fa67fc640 10 monclient: _finish_auth 0 2024-03-29T11:25:24.566 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.580+0000 7f4fa67fc640 10 monclient: _check_auth_tickets 2024-03-29T11:25:24.566 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.580+0000 7f4fa67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:24.566 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.580+0000 7f4fa67fc640 1 -- 172.21.15.114:0/2350089989 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f4fa0001a10 con 0x7f4fa8070550 2024-03-29T11:25:24.566 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.580+0000 7f4fa67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:24.566 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.580+0000 7f4fa67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:25:24.566 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.580+0000 7f4fa67fc640 10 monclient: dump: 2024-03-29T11:25:24.566 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:25:24.566 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:25:24.566 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:25:24.566 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:25:24.566 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:24.566 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:25:24.566 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:25:24.567 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:25:24.567 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:25:24.567 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:25:24.567 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.580+0000 7f4fae139640 5 monclient: authenticate success, global_id 4393 2024-03-29T11:25:24.567 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.580+0000 7f4fae139640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:24.567 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.580+0000 7f4fa67fc640 1 -- 172.21.15.114:0/2350089989 <== mon.1 v1:172.21.15.176:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4fa0001c30 con 0x7f4fa8070550 2024-03-29T11:25:24.567 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.580+0000 7f4fa67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:24.567 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.580+0000 7f4fa67fc640 1 -- 172.21.15.114:0/2350089989 <== mon.1 v1:172.21.15.176:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f4fa0001fc0 con 0x7f4fa8070550 2024-03-29T11:25:24.567 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.580+0000 7f4fa67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:24.567 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.580+0000 7f4fa67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:25:24.567 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.580+0000 7f4fa67fc640 10 monclient: dump: 2024-03-29T11:25:24.567 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:25:24.567 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:25:24.567 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:25:24.567 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:25:24.568 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:24.568 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:25:24.568 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:25:24.568 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:25:24.568 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:25:24.568 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:25:24.568 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.580+0000 7f4fae139640 10 monclient: get_monmap_and_config success 2024-03-29T11:25:24.568 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.580+0000 7f4fae139640 4 set_mon_vals no callback set 2024-03-29T11:25:24.568 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.581+0000 7f4fae139640 10 monclient: shutdown 2024-03-29T11:25:24.568 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.581+0000 7f4fae139640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:24.568 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.581+0000 7f4fae139640 1 -- 172.21.15.114:0/2350089989 >> v1:172.21.15.176:6789/0 conn(0x7f4fa8070550 legacy=0x7f4fa814b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:24.576 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.582+0000 7f4fae139640 1 -- 172.21.15.114:0/2350089989 shutdown_connections 2024-03-29T11:25:24.576 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.582+0000 7f4fae139640 1 -- 172.21.15.114:0/2350089989 >> 172.21.15.114:0/2350089989 conn(0x7f4fa8067610 msgr2=0x7f4fa80b5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:24.576 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.582+0000 7f4fae139640 1 -- 172.21.15.114:0/2350089989 shutdown_connections 2024-03-29T11:25:24.576 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.582+0000 7f4fae139640 1 -- 172.21.15.114:0/2350089989 wait complete. 2024-03-29T11:25:24.576 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.583+0000 7f4fae139640 10 monclient: build_initial_monmap 2024-03-29T11:25:24.576 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.583+0000 7f4fae139640 10 monclient: monmap: 2024-03-29T11:25:24.576 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:25:24.576 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:25:24.577 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T11:25:24.577 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T11:25:24.577 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:24.577 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:25:24.577 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:25:24.577 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T11:25:24.577 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T11:25:24.577 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:25:24.577 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.583+0000 7f4fae139640 1 librados: starting msgr at 2024-03-29T11:25:24.577 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.583+0000 7f4fae139640 1 librados: starting objecter 2024-03-29T11:25:24.577 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.583+0000 7f4fae139640 1 Processor -- start 2024-03-29T11:25:24.577 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.583+0000 7f4fae139640 1 -- start start 2024-03-29T11:25:24.577 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.583+0000 7f4fae139640 1 librados: setting wanted keys 2024-03-29T11:25:24.577 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.583+0000 7f4fae139640 1 librados: calling monclient init 2024-03-29T11:25:24.577 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.583+0000 7f4fae139640 10 monclient: init 2024-03-29T11:25:24.578 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.584+0000 7f4fae139640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:24.578 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.584+0000 7f4fae139640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:25:24.578 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.584+0000 7f4fae139640 10 monclient(hunting): picked mon.noname-a con 0x7f4fa8070170 addr v1:172.21.15.114:6789/0 2024-03-29T11:25:24.578 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.584+0000 7f4fa77fe640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f4fa8070170 0x7f4fa8137ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:40520/0 (socket says 172.21.15.114:40520) 2024-03-29T11:25:24.578 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.584+0000 7f4fa77fe640 1 -- 172.21.15.114:0/1652733213 learned_addr learned my addr 172.21.15.114:0/1652733213 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:25:24.578 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.584+0000 7f4fae139640 10 monclient(hunting): picked mon.noname-c con 0x7f4fa8070550 addr v1:172.21.15.114:6790/0 2024-03-29T11:25:24.578 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.584+0000 7f4f8ffff640 10 client.?.objecter ms_handle_connect 0x7f4fa8070170 2024-03-29T11:25:24.578 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.584+0000 7f4f8ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:24.578 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.584+0000 7f4fae139640 10 monclient(hunting): picked mon.noname-b con 0x7f4fa80713f0 addr v1:172.21.15.176:6789/0 2024-03-29T11:25:24.578 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.585+0000 7f4fae139640 1 -- 172.21.15.114:0/1652733213 --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fa8138c50 con 0x7f4fa8070170 2024-03-29T11:25:24.578 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.585+0000 7f4fae139640 1 -- 172.21.15.114:0/1652733213 --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fa8138f30 con 0x7f4fa80713f0 2024-03-29T11:25:24.578 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.585+0000 7f4fae139640 1 -- 172.21.15.114:0/1652733213 --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fa813f210 con 0x7f4fa8070550 2024-03-29T11:25:24.578 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.585+0000 7f4fae139640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:24.578 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.585+0000 7f4fae139640 10 monclient(hunting): authenticate will time out at 1194.767209s 2024-03-29T11:25:24.579 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.585+0000 7f4f8ffff640 10 client.?.objecter ms_handle_connect 0x7f4fa8070550 2024-03-29T11:25:24.579 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.585+0000 7f4f8ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:24.579 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.585+0000 7f4f8ffff640 10 client.?.objecter ms_handle_connect 0x7f4fa80713f0 2024-03-29T11:25:24.579 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.585+0000 7f4f8ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:24.579 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.585+0000 7f4f8ffff640 1 -- 172.21.15.114:0/1652733213 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2387820703 0 0) 0x7f4fa8138c50 con 0x7f4fa8070170 2024-03-29T11:25:24.579 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.585+0000 7f4f8ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:25:24.579 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.585+0000 7f4f8ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:24.579 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.585+0000 7f4f8ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:24.579 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.585+0000 7f4f8ffff640 10 monclient(hunting): my global_id is 4389 2024-03-29T11:25:24.579 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.585+0000 7f4f8ffff640 1 -- 172.21.15.114:0/1652733213 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f94004eb0 con 0x7f4fa8070170 2024-03-29T11:25:24.579 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.585+0000 7f4f8ffff640 1 -- 172.21.15.114:0/1652733213 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2702276514 0 0) 0x7f4fa813f210 con 0x7f4fa8070550 2024-03-29T11:25:24.579 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.585+0000 7f4f8ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:25:24.579 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.585+0000 7f4f8ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:24.579 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.585+0000 7f4f8ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:24.579 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.585+0000 7f4f8ffff640 10 monclient(hunting): my global_id is 4406 2024-03-29T11:25:24.579 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.585+0000 7f4f8ffff640 1 -- 172.21.15.114:0/1652733213 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4fa8138c50 con 0x7f4fa8070550 2024-03-29T11:25:24.579 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.585+0000 7f4f8ffff640 1 -- 172.21.15.114:0/1652733213 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2547554712 0 0) 0x7f4fa8138f30 con 0x7f4fa80713f0 2024-03-29T11:25:24.580 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.585+0000 7f4f8ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:25:24.580 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.585+0000 7f4f8ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:24.580 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.585+0000 7f4f8ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:24.580 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.585+0000 7f4f8ffff640 10 monclient(hunting): my global_id is 4396 2024-03-29T11:25:24.580 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.585+0000 7f4f8ffff640 1 -- 172.21.15.114:0/1652733213 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4fa813f210 con 0x7f4fa80713f0 2024-03-29T11:25:24.580 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.585+0000 7f4f8ffff640 1 -- 172.21.15.114:0/1652733213 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1362570658 0 0) 0x7f4f94004eb0 con 0x7f4fa8070170 2024-03-29T11:25:24.580 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.585+0000 7f4f8ffff640 1 -- 172.21.15.114:0/1652733213 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4fa8138f30 con 0x7f4fa8070170 2024-03-29T11:25:24.580 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.585+0000 7f4f8ffff640 1 -- 172.21.15.114:0/1652733213 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f4f9c001750 con 0x7f4fa8070170 2024-03-29T11:25:24.580 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.585+0000 7f4f8ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:24.580 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.585+0000 7f4f8ffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:25:24.580 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.585+0000 7f4f8ffff640 10 monclient(hunting): dump: 2024-03-29T11:25:24.580 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:25:24.580 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:25:24.580 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:25:24.580 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:25:24.580 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:24.581 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:25:24.581 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:25:24.581 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:25:24.581 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:25:24.581 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:25:24.581 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.585+0000 7f4f8ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:25:24.581 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.586+0000 7f4f8ffff640 1 -- 172.21.15.114:0/1652733213 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3877692765 0 0) 0x7f4fa813f210 con 0x7f4fa80713f0 2024-03-29T11:25:24.581 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.586+0000 7f4f8ffff640 1 -- 172.21.15.114:0/1652733213 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4f94004eb0 con 0x7f4fa80713f0 2024-03-29T11:25:24.581 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.586+0000 7f4f8ffff640 1 -- 172.21.15.114:0/1652733213 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f4fa0001950 con 0x7f4fa80713f0 2024-03-29T11:25:24.581 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.586+0000 7f4f8ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:24.581 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.586+0000 7f4f8ffff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:25:24.581 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.586+0000 7f4f8ffff640 10 monclient(hunting): dump: 2024-03-29T11:25:24.581 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:25:24.581 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:25:24.582 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:25:24.582 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:25:24.582 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:24.582 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:25:24.582 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:25:24.582 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:25:24.582 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:25:24.582 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:25:24.582 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.586+0000 7f4f8ffff640 1 -- 172.21.15.114:0/1652733213 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1296196992 0 0) 0x7f4fa8138c50 con 0x7f4fa8070550 2024-03-29T11:25:24.582 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.586+0000 7f4f8ffff640 1 -- 172.21.15.114:0/1652733213 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4fa813f210 con 0x7f4fa8070550 2024-03-29T11:25:24.582 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.586+0000 7f4f8ffff640 1 -- 172.21.15.114:0/1652733213 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f4f980018b0 con 0x7f4fa8070550 2024-03-29T11:25:24.582 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.586+0000 7f4f8ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:24.582 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.586+0000 7f4f8ffff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:25:24.582 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.586+0000 7f4f8ffff640 10 monclient(hunting): dump: 2024-03-29T11:25:24.583 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:25:24.583 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:25:24.583 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:25:24.583 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:25:24.583 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:24.583 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:25:24.583 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:25:24.583 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:25:24.583 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:25:24.583 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:25:24.583 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.586+0000 7f4f8ffff640 1 -- 172.21.15.114:0/1652733213 <== mon.1 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4163089824 0 0) 0x7f4f94004eb0 con 0x7f4fa80713f0 2024-03-29T11:25:24.583 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.586+0000 7f4f8ffff640 1 -- 172.21.15.114:0/1652733213 >> v1:172.21.15.114:6790/0 conn(0x7f4fa8070550 legacy=0x7f4fa813c210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:24.583 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.586+0000 7f4f8ffff640 1 -- 172.21.15.114:0/1652733213 >> v1:172.21.15.114:6789/0 conn(0x7f4fa8070170 legacy=0x7f4fa8137ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:24.583 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.586+0000 7f4f8ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:25:24.583 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.586+0000 7f4f8ffff640 1 monclient: found mon.b 2024-03-29T11:25:24.583 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.586+0000 7f4f8ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:24.583 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.586+0000 7f4f8ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:25:24.584 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.586+0000 7f4f8ffff640 1 -- 172.21.15.114:0/1652733213 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fa813aa80 con 0x7f4fa80713f0 2024-03-29T11:25:24.584 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.586+0000 7f4f8ffff640 10 monclient: _finish_auth 0 2024-03-29T11:25:24.584 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.586+0000 7f4f8ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:25:24.584 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.586+0000 7f4f8ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:24.584 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.587+0000 7f4fae139640 5 monclient: authenticate success, global_id 4396 2024-03-29T11:25:24.584 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.587+0000 7f4fae139640 10 monclient: _renew_subs 2024-03-29T11:25:24.584 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.587+0000 7f4fae139640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:25:24.584 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.587+0000 7f4fae139640 1 -- 172.21.15.114:0/1652733213 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fa813eec0 con 0x7f4fa80713f0 2024-03-29T11:25:24.584 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.587+0000 7f4fae139640 10 client.4396.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:25:24.584 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.587+0000 7f4fae139640 10 monclient: _renew_subs 2024-03-29T11:25:24.584 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.587+0000 7f4fae139640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:25:24.584 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.587+0000 7f4fae139640 1 -- 172.21.15.114:0/1652733213 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fa8131480 con 0x7f4fa80713f0 2024-03-29T11:25:24.584 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.587+0000 7f4fae139640 1 librados: init done 2024-03-29T11:25:24.584 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.587+0000 7f4f8ffff640 1 -- 172.21.15.114:0/1652733213 <== mon.1 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4fa00012e0 con 0x7f4fa80713f0 2024-03-29T11:25:24.584 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.588+0000 7f4f8ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:24.585 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.588+0000 7f4fa5ffb640 4 set_mon_vals no callback set 2024-03-29T11:25:24.585 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.588+0000 7f4f8ffff640 1 -- 172.21.15.114:0/1652733213 <== mon.1 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f4fa0003dc0 con 0x7f4fa80713f0 2024-03-29T11:25:24.585 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.588+0000 7f4f8ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:24.585 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.588+0000 7f4fae139640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:24.585 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.588+0000 7f4f8ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:25:24.585 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.588+0000 7f4f8ffff640 10 monclient: dump: 2024-03-29T11:25:24.585 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:25:24.585 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:25:24.585 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:25:24.585 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:25:24.585 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:24.585 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:25:24.585 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:25:24.585 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:25:24.585 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:25:24.585 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:25:24.586 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.589+0000 7f4fae139640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:24.586 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.589+0000 7f4fae139640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:25:24.586 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.589+0000 7f4fae139640 1 -- 172.21.15.114:0/1652733213 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f88005080 con 0x7f4fa80713f0 2024-03-29T11:25:24.586 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.590+0000 7f4f8ffff640 1 -- 172.21.15.114:0/1652733213 <== mon.1 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1594292461 0 0) 0x7f4fa0019d90 con 0x7f4fa80713f0 2024-03-29T11:25:24.586 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.591+0000 7f4fa77fe640 10 monclient: get_auth_request con 0x7f4f9406e030 auth_method 0 2024-03-29T11:25:24.586 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.591+0000 7f4f8ffff640 1 -- 172.21.15.114:0/1652733213 <== mon.1 v1:172.21.15.176:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3238987313 0 0) 0x7f4fa0083b20 con 0x7f4fa80713f0 2024-03-29T11:25:24.586 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.591+0000 7f4f8ffff640 10 client.4396.objecter ms_dispatch 0x7f4fa8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:25:24.586 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.591+0000 7f4f8ffff640 3 client.4396.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:25:24.586 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.591+0000 7f4f8ffff640 3 client.4396.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:25:24.586 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.591+0000 7f4f8ffff640 20 client.4396.objecter dump_active .. 0 homeless 2024-03-29T11:25:24.586 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.591+0000 7f4f8ffff640 10 client.4396.objecter ms_handle_connect 0x7f4f9406e030 2024-03-29T11:25:24.587 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.599+0000 7f4f8ffff640 1 -- 172.21.15.114:0/1652733213 <== mon.1 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4fa004f400 con 0x7f4fa80713f0 2024-03-29T11:25:24.587 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.599+0000 7f4f8ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:24.587 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.599+0000 7f4f8ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:25:24.888 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.904+0000 7f4fae139640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:24.888 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.904+0000 7f4fae139640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:24.888 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.904+0000 7f4fae139640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:25:24.888 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.904+0000 7f4fae139640 1 -- 172.21.15.114:0/1652733213 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4f88005330 con 0x7f4fa80713f0 2024-03-29T11:25:24.889 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.905+0000 7f4f8ffff640 1 -- 172.21.15.114:0/1652733213 <== mon.1 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2858224613) 0x7f4fa0051330 con 0x7f4fa80713f0 2024-03-29T11:25:24.889 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.905+0000 7f4f8ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:24.889 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.905+0000 7f4f8ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:25:24.889 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T11:25:24.889 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"b0de2b9b-c45a-44c7-b6ee-e83b39a1e108","created":"2024-03-29T11:20:49.420660+0000","modified":"2024-03-29T11:20:51.700167+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:51.700167+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a64785d3-6b11-499a-b706-cc827ab96a74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5882dadb-e7a8-4604-ba9a-418013c106c8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d7e8c409-bc66-4e4a-bbdb-420bad31388e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0682989-3595-4cd6-90a5-ab9111c278e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"67068120-e3ff-4817-bd06-44feb7479a35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09e8b-8a85-4997-bdea-75d754e6ee67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2f0d9ead-f47f-4cb3-92a4-0a5ad23cd831","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fdb6a25f-6bd8-4c65-9b9b-b460ba7673a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:25:24.892 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.908+0000 7f4fae139640 10 librados: watch_flush enter 2024-03-29T11:25:24.892 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.908+0000 7f4fae139640 10 librados: watch_flush exit 2024-03-29T11:25:24.892 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.908+0000 7f4fae139640 20 client.4396.objecter shutdown clearing up homeless session... 2024-03-29T11:25:24.893 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.908+0000 7f4fae139640 10 client.4396.objecter successfully canceled tick 2024-03-29T11:25:24.893 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.908+0000 7f4fae139640 1 -- 172.21.15.114:0/1652733213 >> v1:172.21.15.114:6816/4038755055 conn(0x7f4f9406e030 legacy=0x7f4f940704f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:24.893 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.908+0000 7f4fae139640 10 monclient: shutdown 2024-03-29T11:25:24.893 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.908+0000 7f4fae139640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:24.893 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.908+0000 7f4fae139640 1 -- 172.21.15.114:0/1652733213 >> v1:172.21.15.176:6789/0 conn(0x7f4fa80713f0 legacy=0x7f4fa81dfd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:24.893 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.908+0000 7f4fae139640 1 -- 172.21.15.114:0/1652733213 shutdown_connections 2024-03-29T11:25:24.893 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.909+0000 7f4fae139640 1 -- 172.21.15.114:0/1652733213 >> 172.21.15.114:0/1652733213 conn(0x7f4fa8067610 msgr2=0x7f4fa81406e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:24.893 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.909+0000 7f4fae139640 1 -- 172.21.15.114:0/1652733213 shutdown_connections 2024-03-29T11:25:24.894 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.909+0000 7f4fae139640 1 -- 172.21.15.114:0/1652733213 wait complete. 2024-03-29T11:25:24.894 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:24.909+0000 7f4fae139640 1 librados: shutdown 2024-03-29T11:25:24.906 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:25:31.713 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:25:31.865 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.880+0000 7fa52659a640 10 monclient: get_monmap_and_config 2024-03-29T11:25:31.865 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.880+0000 7fa52659a640 10 monclient: build_initial_monmap 2024-03-29T11:25:31.865 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.880+0000 7fa52659a640 10 monclient: monmap: 2024-03-29T11:25:31.865 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:25:31.866 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:25:31.866 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:25:31.881252+0000 2024-03-29T11:25:31.866 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:25:31.881252+0000 2024-03-29T11:25:31.866 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:31.866 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:25:31.866 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:25:31.866 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T11:25:31.866 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T11:25:31.866 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:25:31.866 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.881+0000 7fa52659a640 1 Processor -- start 2024-03-29T11:25:31.866 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.881+0000 7fa52659a640 1 -- start start 2024-03-29T11:25:31.866 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.881+0000 7fa52659a640 10 monclient: init 2024-03-29T11:25:31.867 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.882+0000 7fa52659a640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:31.867 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.882+0000 7fa52659a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:25:31.867 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.882+0000 7fa52659a640 10 monclient(hunting): picked mon.noname-c con 0x7fa5200a9ac0 addr v1:172.21.15.176:6789/0 2024-03-29T11:25:31.867 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.882+0000 7fa52659a640 10 monclient(hunting): picked mon.noname-b con 0x7fa52013ee90 addr v1:172.21.15.114:6790/0 2024-03-29T11:25:31.867 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.882+0000 7fa52659a640 10 monclient(hunting): picked mon.noname-a con 0x7fa5200af900 addr v1:172.21.15.114:6789/0 2024-03-29T11:25:31.867 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.882+0000 7fa52659a640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5200afd10 con 0x7fa5200af900 2024-03-29T11:25:31.867 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.882+0000 7fa52659a640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5200af6f0 con 0x7fa5200a9ac0 2024-03-29T11:25:31.867 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.882+0000 7fa52659a640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5200aac80 con 0x7fa52013ee90 2024-03-29T11:25:31.868 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.882+0000 7fa52659a640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:31.868 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.882+0000 7fa52659a640 10 monclient(hunting): authenticate will time out at 1202.064692s 2024-03-29T11:25:31.868 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.882+0000 7fa51f7fe640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7fa52013ee90 0x7fa52013b5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:58076/0 (socket says 172.21.15.114:58076) 2024-03-29T11:25:31.868 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.882+0000 7fa51f7fe640 1 -- 172.21.15.114:0/2395864220 learned_addr learned my addr 172.21.15.114:0/2395864220 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:25:31.868 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.883+0000 7fa51effd640 1 -- 172.21.15.114:0/2395864220 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 318454335 0 0) 0x7fa5200aac80 con 0x7fa52013ee90 2024-03-29T11:25:31.868 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.883+0000 7fa51effd640 10 monclient(hunting): _negotiate 2024-03-29T11:25:31.868 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.883+0000 7fa51effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:31.868 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.883+0000 7fa51effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:31.868 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.883+0000 7fa51effd640 10 monclient(hunting): my global_id is 4409 2024-03-29T11:25:31.868 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.883+0000 7fa51effd640 1 -- 172.21.15.114:0/2395864220 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa508003520 con 0x7fa52013ee90 2024-03-29T11:25:31.868 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.883+0000 7fa51effd640 1 -- 172.21.15.114:0/2395864220 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2274384552 0 0) 0x7fa508003520 con 0x7fa52013ee90 2024-03-29T11:25:31.869 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.883+0000 7fa51effd640 1 -- 172.21.15.114:0/2395864220 >> v1:172.21.15.176:6789/0 conn(0x7fa5200a9ac0 legacy=0x7fa5200adfe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:25:31.869 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.883+0000 7fa51effd640 1 -- 172.21.15.114:0/2395864220 >> v1:172.21.15.114:6789/0 conn(0x7fa5200af900 legacy=0x7fa520149b00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:25:31.870 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.885+0000 7fa51effd640 10 monclient: _finish_hunting 0 2024-03-29T11:25:31.870 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.885+0000 7fa51effd640 1 monclient: found mon.noname-b 2024-03-29T11:25:31.871 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.885+0000 7fa51effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:31.871 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.885+0000 7fa51effd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.114:6790/0 2024-03-29T11:25:31.871 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.885+0000 7fa51effd640 1 -- 172.21.15.114:0/2395864220 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5200aaf60 con 0x7fa52013ee90 2024-03-29T11:25:31.871 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.885+0000 7fa51effd640 10 monclient: _finish_auth 0 2024-03-29T11:25:31.871 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.885+0000 7fa51effd640 10 monclient: _check_auth_tickets 2024-03-29T11:25:31.871 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.885+0000 7fa51effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:31.871 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.885+0000 7fa51effd640 1 -- 172.21.15.114:0/2395864220 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fa514001670 con 0x7fa52013ee90 2024-03-29T11:25:31.871 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.885+0000 7fa51effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:31.871 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.885+0000 7fa51effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:25:31.871 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.885+0000 7fa51effd640 10 monclient: dump: 2024-03-29T11:25:31.871 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:25:31.871 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:25:31.871 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:25:31.871 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:25:31.871 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:31.871 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:25:31.872 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:25:31.872 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:25:31.872 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:25:31.872 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:25:31.872 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.885+0000 7fa52659a640 5 monclient: authenticate success, global_id 4409 2024-03-29T11:25:31.872 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.885+0000 7fa52659a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:31.872 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.886+0000 7fa51effd640 1 -- 172.21.15.114:0/2395864220 <== mon.2 v1:172.21.15.114:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa5140018d0 con 0x7fa52013ee90 2024-03-29T11:25:31.872 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.886+0000 7fa51effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:31.873 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.886+0000 7fa52659a640 10 monclient: get_monmap_and_config success 2024-03-29T11:25:31.873 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.886+0000 7fa52659a640 4 set_mon_vals no callback set 2024-03-29T11:25:31.873 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.887+0000 7fa52659a640 10 monclient: shutdown 2024-03-29T11:25:31.873 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.887+0000 7fa52659a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:31.873 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.887+0000 7fa52659a640 1 -- 172.21.15.114:0/2395864220 >> v1:172.21.15.114:6790/0 conn(0x7fa52013ee90 legacy=0x7fa52013b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:31.873 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.887+0000 7fa51effd640 1 -- 172.21.15.114:0/2395864220 <== mon.2 v1:172.21.15.114:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fa514001d20 con 0x7fa52013ee90 2024-03-29T11:25:31.873 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.887+0000 7fa51effd640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-29T11:25:31.873 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.887+0000 7fa52659a640 1 -- 172.21.15.114:0/2395864220 shutdown_connections 2024-03-29T11:25:31.874 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.887+0000 7fa52659a640 1 -- 172.21.15.114:0/2395864220 >> 172.21.15.114:0/2395864220 conn(0x7fa5200b35e0 msgr2=0x7fa5200b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:31.874 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.887+0000 7fa52659a640 1 -- 172.21.15.114:0/2395864220 shutdown_connections 2024-03-29T11:25:31.875 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.887+0000 7fa52659a640 1 -- 172.21.15.114:0/2395864220 wait complete. 2024-03-29T11:25:31.875 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.888+0000 7fa52659a640 10 monclient: build_initial_monmap 2024-03-29T11:25:31.875 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.888+0000 7fa52659a640 10 monclient: monmap: 2024-03-29T11:25:31.876 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:25:31.876 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:25:31.876 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T11:25:31.876 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T11:25:31.876 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:31.876 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:25:31.876 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:25:31.876 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T11:25:31.876 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T11:25:31.876 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:25:31.876 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.889+0000 7fa52659a640 1 librados: starting msgr at 2024-03-29T11:25:31.876 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.889+0000 7fa52659a640 1 librados: starting objecter 2024-03-29T11:25:31.876 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.889+0000 7fa52659a640 1 Processor -- start 2024-03-29T11:25:31.876 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.889+0000 7fa52659a640 1 -- start start 2024-03-29T11:25:31.877 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.889+0000 7fa52659a640 1 librados: setting wanted keys 2024-03-29T11:25:31.877 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.889+0000 7fa52659a640 1 librados: calling monclient init 2024-03-29T11:25:31.877 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.889+0000 7fa52659a640 10 monclient: init 2024-03-29T11:25:31.877 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.890+0000 7fa52659a640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:31.877 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.890+0000 7fa52659a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:25:31.877 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.890+0000 7fa52659a640 10 monclient(hunting): picked mon.noname-b con 0x7fa5200a9ac0 addr v1:172.21.15.176:6789/0 2024-03-29T11:25:31.877 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.890+0000 7fa52659a640 10 monclient(hunting): picked mon.noname-c con 0x7fa5200af900 addr v1:172.21.15.114:6790/0 2024-03-29T11:25:31.877 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.890+0000 7fa52659a640 10 monclient(hunting): picked mon.noname-a con 0x7fa52013ee90 addr v1:172.21.15.114:6789/0 2024-03-29T11:25:31.877 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.890+0000 7fa52659a640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5200707b0 con 0x7fa52013ee90 2024-03-29T11:25:31.877 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.890+0000 7fa52659a640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5201d6150 con 0x7fa5200a9ac0 2024-03-29T11:25:31.877 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.890+0000 7fa52659a640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5201d62e0 con 0x7fa5200af900 2024-03-29T11:25:31.877 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.890+0000 7fa52659a640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:31.877 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.890+0000 7fa52659a640 10 monclient(hunting): authenticate will time out at 1202.072476s 2024-03-29T11:25:31.878 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.894+0000 7fa524b37640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fa52013ee90 0x7fa5201de4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:43083/0 (socket says 172.21.15.114:43083) 2024-03-29T11:25:31.878 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.894+0000 7fa524b37640 1 -- 172.21.15.114:0/762399735 learned_addr learned my addr 172.21.15.114:0/762399735 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:25:31.879 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.895+0000 7fa51cff9640 10 client.?.objecter ms_handle_connect 0x7fa52013ee90 2024-03-29T11:25:31.879 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.895+0000 7fa51cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:31.879 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.895+0000 7fa51cff9640 1 -- 172.21.15.114:0/762399735 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 721212468 0 0) 0x7fa5200707b0 con 0x7fa52013ee90 2024-03-29T11:25:31.879 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.895+0000 7fa51cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:25:31.879 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.895+0000 7fa51cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:31.881 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.895+0000 7fa51cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:31.881 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.895+0000 7fa51cff9640 10 monclient(hunting): my global_id is 4392 2024-03-29T11:25:31.882 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.896+0000 7fa51cff9640 1 -- 172.21.15.114:0/762399735 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa50c004eb0 con 0x7fa52013ee90 2024-03-29T11:25:31.882 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.896+0000 7fa51cff9640 10 client.?.objecter ms_handle_connect 0x7fa5200af900 2024-03-29T11:25:31.882 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.896+0000 7fa51cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:31.882 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.896+0000 7fa51cff9640 1 -- 172.21.15.114:0/762399735 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1037494399 0 0) 0x7fa5201d62e0 con 0x7fa5200af900 2024-03-29T11:25:31.882 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.896+0000 7fa51cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:25:31.882 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.896+0000 7fa51cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:31.882 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.896+0000 7fa51cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:31.882 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.896+0000 7fa51cff9640 10 monclient(hunting): my global_id is 4412 2024-03-29T11:25:31.882 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.896+0000 7fa51cff9640 1 -- 172.21.15.114:0/762399735 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5200707b0 con 0x7fa5200af900 2024-03-29T11:25:31.882 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.896+0000 7fa51cff9640 1 -- 172.21.15.114:0/762399735 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2295289709 0 0) 0x7fa50c004eb0 con 0x7fa52013ee90 2024-03-29T11:25:31.882 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.896+0000 7fa51cff9640 1 -- 172.21.15.114:0/762399735 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa5201d62e0 con 0x7fa52013ee90 2024-03-29T11:25:31.882 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.896+0000 7fa51cff9640 1 -- 172.21.15.114:0/762399735 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fa5100018e0 con 0x7fa52013ee90 2024-03-29T11:25:31.882 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.896+0000 7fa51cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:31.883 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.896+0000 7fa51cff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:25:31.883 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.897+0000 7fa51cff9640 10 monclient(hunting): dump: 2024-03-29T11:25:31.883 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:25:31.883 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:25:31.883 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:25:31.883 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:25:31.883 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:31.883 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:25:31.883 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:25:31.883 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:25:31.883 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:25:31.883 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:25:31.883 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.897+0000 7fa51cff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:25:31.883 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.897+0000 7fa51cff9640 1 -- 172.21.15.114:0/762399735 <== mon.0 v1:172.21.15.114:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3782732009 0 0) 0x7fa5201d62e0 con 0x7fa52013ee90 2024-03-29T11:25:31.883 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.897+0000 7fa51cff9640 1 -- 172.21.15.114:0/762399735 >> v1:172.21.15.114:6790/0 conn(0x7fa5200af900 legacy=0x7fa5201396f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:31.883 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.897+0000 7fa51cff9640 1 -- 172.21.15.114:0/762399735 >> v1:172.21.15.176:6789/0 conn(0x7fa5200a9ac0 legacy=0x7fa5201d6a50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:25:31.887 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.898+0000 7fa51cff9640 10 monclient: _finish_hunting 0 2024-03-29T11:25:31.887 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.898+0000 7fa51cff9640 1 monclient: found mon.a 2024-03-29T11:25:31.887 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.898+0000 7fa51cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:31.887 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.898+0000 7fa51cff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T11:25:31.887 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.898+0000 7fa51cff9640 1 -- 172.21.15.114:0/762399735 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa52006ea10 con 0x7fa52013ee90 2024-03-29T11:25:31.887 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.898+0000 7fa51cff9640 10 monclient: _finish_auth 0 2024-03-29T11:25:31.887 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.898+0000 7fa51cff9640 10 monclient: _check_auth_tickets 2024-03-29T11:25:31.887 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.898+0000 7fa51cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:31.887 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.898+0000 7fa52659a640 5 monclient: authenticate success, global_id 4392 2024-03-29T11:25:31.887 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.898+0000 7fa52659a640 10 monclient: _renew_subs 2024-03-29T11:25:31.888 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.898+0000 7fa52659a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T11:25:31.888 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.898+0000 7fa52659a640 1 -- 172.21.15.114:0/762399735 --> v1:172.21.15.114:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa52006ec60 con 0x7fa52013ee90 2024-03-29T11:25:31.888 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.898+0000 7fa52659a640 10 client.4392.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:25:31.888 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.898+0000 7fa52659a640 10 monclient: _renew_subs 2024-03-29T11:25:31.888 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.898+0000 7fa52659a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T11:25:31.888 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.898+0000 7fa52659a640 1 -- 172.21.15.114:0/762399735 --> v1:172.21.15.114:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa520139fe0 con 0x7fa52013ee90 2024-03-29T11:25:31.888 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.899+0000 7fa52659a640 1 librados: init done 2024-03-29T11:25:31.888 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.899+0000 7fa51cff9640 1 -- 172.21.15.114:0/762399735 <== mon.0 v1:172.21.15.114:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa5100035f0 con 0x7fa52013ee90 2024-03-29T11:25:31.888 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.899+0000 7fa51cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:31.888 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.899+0000 7fa51e7fc640 4 set_mon_vals no callback set 2024-03-29T11:25:31.888 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.900+0000 7fa51cff9640 1 -- 172.21.15.114:0/762399735 <== mon.0 v1:172.21.15.114:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fa5100039c0 con 0x7fa52013ee90 2024-03-29T11:25:31.888 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.900+0000 7fa51cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:31.888 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.900+0000 7fa51cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:25:31.888 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.900+0000 7fa51cff9640 10 monclient: dump: 2024-03-29T11:25:31.888 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:25:31.889 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:25:31.889 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:25:31.889 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:25:31.889 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:31.889 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:25:31.889 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:25:31.889 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:25:31.889 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:25:31.889 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:25:31.889 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.900+0000 7fa52659a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:31.889 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.901+0000 7fa52659a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:31.889 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.901+0000 7fa52659a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T11:25:31.889 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.901+0000 7fa52659a640 1 -- 172.21.15.114:0/762399735 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5201d68a0 con 0x7fa52013ee90 2024-03-29T11:25:31.889 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.902+0000 7fa51cff9640 1 -- 172.21.15.114:0/762399735 <== mon.0 v1:172.21.15.114:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1594292461 0 0) 0x7fa510003b20 con 0x7fa52013ee90 2024-03-29T11:25:31.889 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.902+0000 7fa51cff9640 1 -- 172.21.15.114:0/762399735 <== mon.0 v1:172.21.15.114:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3238987313 0 0) 0x7fa5100849f0 con 0x7fa52013ee90 2024-03-29T11:25:31.889 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.902+0000 7fa51cff9640 10 client.4392.objecter ms_dispatch 0x7fa520000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:25:31.890 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.902+0000 7fa51cff9640 3 client.4392.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:25:31.890 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.902+0000 7fa51cff9640 3 client.4392.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:25:31.890 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.902+0000 7fa51cff9640 20 client.4392.objecter dump_active .. 0 homeless 2024-03-29T11:25:31.890 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.903+0000 7fa51ffff640 10 monclient: get_auth_request con 0x7fa50c06dd00 auth_method 0 2024-03-29T11:25:31.890 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.903+0000 7fa51cff9640 10 client.4392.objecter ms_handle_connect 0x7fa50c06dd00 2024-03-29T11:25:31.895 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.911+0000 7fa51cff9640 1 -- 172.21.15.114:0/762399735 <== mon.0 v1:172.21.15.114:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa5100502d0 con 0x7fa52013ee90 2024-03-29T11:25:31.895 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.911+0000 7fa51cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:31.895 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:31.911+0000 7fa51cff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:25:32.210 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:32.225+0000 7fa52659a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:32.210 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:32.225+0000 7fa52659a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:32.210 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:32.225+0000 7fa52659a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T11:25:32.210 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:32.225+0000 7fa52659a640 1 -- 172.21.15.114:0/762399735 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa52013a280 con 0x7fa52013ee90 2024-03-29T11:25:32.210 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:32.226+0000 7fa51cff9640 1 -- 172.21.15.114:0/762399735 <== mon.0 v1:172.21.15.114:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2858224613) 0x7fa52013a280 con 0x7fa52013ee90 2024-03-29T11:25:32.210 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:32.226+0000 7fa51cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:32.211 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:32.226+0000 7fa51cff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:25:32.211 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T11:25:32.211 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"b0de2b9b-c45a-44c7-b6ee-e83b39a1e108","created":"2024-03-29T11:20:49.420660+0000","modified":"2024-03-29T11:20:51.700167+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:51.700167+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a64785d3-6b11-499a-b706-cc827ab96a74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5882dadb-e7a8-4604-ba9a-418013c106c8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d7e8c409-bc66-4e4a-bbdb-420bad31388e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0682989-3595-4cd6-90a5-ab9111c278e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"67068120-e3ff-4817-bd06-44feb7479a35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09e8b-8a85-4997-bdea-75d754e6ee67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2f0d9ead-f47f-4cb3-92a4-0a5ad23cd831","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fdb6a25f-6bd8-4c65-9b9b-b460ba7673a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:25:32.213 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:32.228+0000 7fa52659a640 10 librados: watch_flush enter 2024-03-29T11:25:32.213 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:32.229+0000 7fa52659a640 10 librados: watch_flush exit 2024-03-29T11:25:32.213 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:32.229+0000 7fa52659a640 20 client.4392.objecter shutdown clearing up homeless session... 2024-03-29T11:25:32.213 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:32.229+0000 7fa52659a640 10 client.4392.objecter successfully canceled tick 2024-03-29T11:25:32.213 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:32.229+0000 7fa52659a640 1 -- 172.21.15.114:0/762399735 >> v1:172.21.15.114:6816/4038755055 conn(0x7fa50c06dd00 legacy=0x7fa50c0701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:32.213 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:32.229+0000 7fa52659a640 10 monclient: shutdown 2024-03-29T11:25:32.213 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:32.229+0000 7fa52659a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:32.213 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:32.229+0000 7fa52659a640 1 -- 172.21.15.114:0/762399735 >> v1:172.21.15.114:6789/0 conn(0x7fa52013ee90 legacy=0x7fa5201de4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:32.213 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:32.229+0000 7fa52659a640 1 -- 172.21.15.114:0/762399735 shutdown_connections 2024-03-29T11:25:32.214 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:32.229+0000 7fa52659a640 1 -- 172.21.15.114:0/762399735 >> 172.21.15.114:0/762399735 conn(0x7fa5200b35e0 msgr2=0x7fa52013b0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:32.214 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:32.230+0000 7fa52659a640 1 -- 172.21.15.114:0/762399735 shutdown_connections 2024-03-29T11:25:32.214 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:32.230+0000 7fa52659a640 1 -- 172.21.15.114:0/762399735 wait complete. 2024-03-29T11:25:32.214 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:32.230+0000 7fa52659a640 1 librados: shutdown 2024-03-29T11:25:32.227 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:25:39.034 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:25:39.187 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.201+0000 7f0ff2af1640 10 monclient: get_monmap_and_config 2024-03-29T11:25:39.187 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.201+0000 7f0ff2af1640 10 monclient: build_initial_monmap 2024-03-29T11:25:39.187 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.201+0000 7f0ff2af1640 10 monclient: monmap: 2024-03-29T11:25:39.187 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:25:39.187 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:25:39.187 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:25:39.202522+0000 2024-03-29T11:25:39.187 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:25:39.202522+0000 2024-03-29T11:25:39.187 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:39.187 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:25:39.188 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:25:39.188 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T11:25:39.188 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T11:25:39.188 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:25:39.188 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.202+0000 7f0ff2af1640 1 Processor -- start 2024-03-29T11:25:39.188 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.202+0000 7f0ff2af1640 1 -- start start 2024-03-29T11:25:39.188 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.202+0000 7f0ff2af1640 10 monclient: init 2024-03-29T11:25:39.189 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.203+0000 7f0ff2af1640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:39.189 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.203+0000 7f0ff2af1640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:25:39.189 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.203+0000 7f0ff2af1640 10 monclient(hunting): picked mon.noname-c con 0x7f0fec1457f0 addr v1:172.21.15.176:6789/0 2024-03-29T11:25:39.189 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.203+0000 7f0ff2af1640 10 monclient(hunting): picked mon.noname-a con 0x7f0fec13f650 addr v1:172.21.15.114:6789/0 2024-03-29T11:25:39.190 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.203+0000 7f0ff2af1640 10 monclient(hunting): picked mon.noname-b con 0x7f0fec0a9d70 addr v1:172.21.15.114:6790/0 2024-03-29T11:25:39.190 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.203+0000 7f0ff2af1640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fec0aaef0 con 0x7f0fec13f650 2024-03-29T11:25:39.190 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.203+0000 7f0ff2af1640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fec145bd0 con 0x7f0fec1457f0 2024-03-29T11:25:39.190 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.203+0000 7f0ff2af1640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fec145eb0 con 0x7f0fec0a9d70 2024-03-29T11:25:39.190 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.203+0000 7f0ff2af1640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:39.190 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.203+0000 7f0ff2af1640 10 monclient(hunting): authenticate will time out at 1209.385778s 2024-03-29T11:25:39.191 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.205+0000 7f0ff108e640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f0fec0a9d70 0x7f0fec14d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:57736/0 (socket says 172.21.15.114:57736) 2024-03-29T11:25:39.191 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.205+0000 7f0ff108e640 1 -- 172.21.15.114:0/1375762555 learned_addr learned my addr 172.21.15.114:0/1375762555 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:25:39.191 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.206+0000 7f0feb7fe640 1 -- 172.21.15.114:0/1375762555 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4200859676 0 0) 0x7f0fec145eb0 con 0x7f0fec0a9d70 2024-03-29T11:25:39.191 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.206+0000 7f0feb7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:25:39.191 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.206+0000 7f0feb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:39.191 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.206+0000 7f0feb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:39.191 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.206+0000 7f0feb7fe640 10 monclient(hunting): my global_id is 4415 2024-03-29T11:25:39.192 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.206+0000 7f0feb7fe640 1 -- 172.21.15.114:0/1375762555 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0fd4003520 con 0x7f0fec0a9d70 2024-03-29T11:25:39.192 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.207+0000 7f0feb7fe640 1 -- 172.21.15.114:0/1375762555 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 102444122 0 0) 0x7f0fd4003520 con 0x7f0fec0a9d70 2024-03-29T11:25:39.192 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.207+0000 7f0feb7fe640 1 -- 172.21.15.114:0/1375762555 >> v1:172.21.15.176:6789/0 conn(0x7f0fec1457f0 legacy=0x7f0fec13ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:25:39.195 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.209+0000 7f0feb7fe640 1 -- 172.21.15.114:0/1375762555 >> v1:172.21.15.114:6789/0 conn(0x7f0fec13f650 legacy=0x7f0fec0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:25:39.195 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.209+0000 7f0feb7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:25:39.195 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.209+0000 7f0feb7fe640 1 monclient: found mon.noname-b 2024-03-29T11:25:39.195 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.209+0000 7f0feb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:39.195 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.209+0000 7f0feb7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.114:6790/0 2024-03-29T11:25:39.195 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.209+0000 7f0feb7fe640 1 -- 172.21.15.114:0/1375762555 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fec146340 con 0x7f0fec0a9d70 2024-03-29T11:25:39.196 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.210+0000 7f0feb7fe640 10 monclient: _finish_auth 0 2024-03-29T11:25:39.196 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.210+0000 7f0feb7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:25:39.196 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.210+0000 7f0feb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:39.196 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.210+0000 7f0ff2af1640 5 monclient: authenticate success, global_id 4415 2024-03-29T11:25:39.196 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.210+0000 7f0ff2af1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:39.196 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.210+0000 7f0feb7fe640 1 -- 172.21.15.114:0/1375762555 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f0fe4001a10 con 0x7f0fec0a9d70 2024-03-29T11:25:39.196 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.210+0000 7f0feb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:39.196 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.210+0000 7f0feb7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:25:39.196 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.210+0000 7f0feb7fe640 10 monclient: dump: 2024-03-29T11:25:39.196 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:25:39.196 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:25:39.196 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:25:39.196 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:25:39.197 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:39.197 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:25:39.197 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:25:39.197 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:25:39.197 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:25:39.197 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:25:39.197 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.210+0000 7f0ff2af1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:39.197 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.210+0000 7f0feb7fe640 1 -- 172.21.15.114:0/1375762555 <== mon.2 v1:172.21.15.114:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0fe4001ca0 con 0x7f0fec0a9d70 2024-03-29T11:25:39.197 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.210+0000 7f0feb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:39.199 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.210+0000 7f0ff2af1640 10 monclient: get_monmap_and_config success 2024-03-29T11:25:39.199 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.210+0000 7f0ff2af1640 4 set_mon_vals no callback set 2024-03-29T11:25:39.199 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.211+0000 7f0feb7fe640 1 -- 172.21.15.114:0/1375762555 <== mon.2 v1:172.21.15.114:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f0fe4002070 con 0x7f0fec0a9d70 2024-03-29T11:25:39.199 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.211+0000 7f0feb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:39.199 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.211+0000 7f0feb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:25:39.199 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.211+0000 7f0feb7fe640 10 monclient: dump: 2024-03-29T11:25:39.199 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:25:39.199 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:25:39.199 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:25:39.199 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:25:39.199 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:39.199 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:25:39.199 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:25:39.199 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:25:39.199 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:25:39.200 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:25:39.200 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.211+0000 7f0ff2af1640 10 monclient: shutdown 2024-03-29T11:25:39.200 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.211+0000 7f0ff2af1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:39.200 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.211+0000 7f0ff2af1640 1 -- 172.21.15.114:0/1375762555 >> v1:172.21.15.114:6790/0 conn(0x7f0fec0a9d70 legacy=0x7f0fec14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:39.200 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.212+0000 7f0ff2af1640 1 -- 172.21.15.114:0/1375762555 shutdown_connections 2024-03-29T11:25:39.200 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.212+0000 7f0ff2af1640 1 -- 172.21.15.114:0/1375762555 >> 172.21.15.114:0/1375762555 conn(0x7f0fec066a00 msgr2=0x7f0fec066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:39.200 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.213+0000 7f0ff2af1640 1 -- 172.21.15.114:0/1375762555 shutdown_connections 2024-03-29T11:25:39.203 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.213+0000 7f0ff2af1640 1 -- 172.21.15.114:0/1375762555 wait complete. 2024-03-29T11:25:39.203 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.214+0000 7f0ff2af1640 10 monclient: build_initial_monmap 2024-03-29T11:25:39.203 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.214+0000 7f0ff2af1640 10 monclient: monmap: 2024-03-29T11:25:39.203 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:25:39.203 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:25:39.203 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T11:25:39.204 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T11:25:39.204 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:39.204 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:25:39.204 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:25:39.204 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T11:25:39.204 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T11:25:39.204 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:25:39.204 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.214+0000 7f0ff2af1640 1 librados: starting msgr at 2024-03-29T11:25:39.204 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.214+0000 7f0ff2af1640 1 librados: starting objecter 2024-03-29T11:25:39.204 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.214+0000 7f0ff2af1640 1 Processor -- start 2024-03-29T11:25:39.204 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.214+0000 7f0ff2af1640 1 -- start start 2024-03-29T11:25:39.204 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.214+0000 7f0ff2af1640 1 librados: setting wanted keys 2024-03-29T11:25:39.204 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.214+0000 7f0ff2af1640 1 librados: calling monclient init 2024-03-29T11:25:39.204 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.214+0000 7f0ff2af1640 10 monclient: init 2024-03-29T11:25:39.204 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.215+0000 7f0ff2af1640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:39.204 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.215+0000 7f0ff2af1640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:25:39.205 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.215+0000 7f0ff2af1640 10 monclient(hunting): picked mon.noname-b con 0x7f0fec0a9d70 addr v1:172.21.15.176:6789/0 2024-03-29T11:25:39.205 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.215+0000 7f0ff2af1640 10 monclient(hunting): picked mon.noname-a con 0x7f0fec13f650 addr v1:172.21.15.114:6789/0 2024-03-29T11:25:39.205 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.215+0000 7f0ff2af1640 10 monclient(hunting): picked mon.noname-c con 0x7f0fec1457f0 addr v1:172.21.15.114:6790/0 2024-03-29T11:25:39.205 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.215+0000 7f0ff2af1640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fec0ac260 con 0x7f0fec13f650 2024-03-29T11:25:39.205 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.215+0000 7f0ff2af1640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fec0ac4d0 con 0x7f0fec0a9d70 2024-03-29T11:25:39.205 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.215+0000 7f0ff2af1640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fec0ac7b0 con 0x7f0fec1457f0 2024-03-29T11:25:39.205 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.215+0000 7f0ff2af1640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:39.205 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.215+0000 7f0ff2af1640 10 monclient(hunting): authenticate will time out at 1209.397767s 2024-03-29T11:25:39.205 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.216+0000 7f0ff108e640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f0fec1457f0 0x7f0fec1f3280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:58250/0 (socket says 172.21.15.114:58250) 2024-03-29T11:25:39.205 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.216+0000 7f0ff108e640 1 -- 172.21.15.114:0/1495865435 learned_addr learned my addr 172.21.15.114:0/1495865435 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:25:39.205 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.217+0000 7f0fe97fa640 10 client.?.objecter ms_handle_connect 0x7f0fec1457f0 2024-03-29T11:25:39.205 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.217+0000 7f0fe97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:39.205 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.217+0000 7f0fe97fa640 1 -- 172.21.15.114:0/1495865435 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 856447735 0 0) 0x7f0fec0ac7b0 con 0x7f0fec1457f0 2024-03-29T11:25:39.205 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.217+0000 7f0fe97fa640 10 monclient(hunting): _negotiate 2024-03-29T11:25:39.206 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.217+0000 7f0fe97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:39.206 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.217+0000 7f0fe97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:39.206 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.217+0000 7f0fe97fa640 10 monclient(hunting): my global_id is 4418 2024-03-29T11:25:39.206 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.217+0000 7f0fe97fa640 1 -- 172.21.15.114:0/1495865435 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0fd8004eb0 con 0x7f0fec1457f0 2024-03-29T11:25:39.209 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.220+0000 7f0fe97fa640 1 -- 172.21.15.114:0/1495865435 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2057204371 0 0) 0x7f0fd8004eb0 con 0x7f0fec1457f0 2024-03-29T11:25:39.209 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.221+0000 7f0fe97fa640 1 -- 172.21.15.114:0/1495865435 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0fec0ac7b0 con 0x7f0fec1457f0 2024-03-29T11:25:39.209 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.221+0000 7f0fe97fa640 1 -- 172.21.15.114:0/1495865435 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f0fe40018d0 con 0x7f0fec1457f0 2024-03-29T11:25:39.209 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.221+0000 7f0fe97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:39.210 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.221+0000 7f0fe97fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:25:39.210 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.221+0000 7f0fe97fa640 10 monclient(hunting): dump: 2024-03-29T11:25:39.210 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:25:39.210 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:25:39.210 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:25:39.210 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:25:39.210 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:39.210 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:25:39.210 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:25:39.210 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:25:39.210 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:25:39.210 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:25:39.210 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.221+0000 7f0fe97fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:25:39.210 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.221+0000 7f0fe97fa640 10 client.?.objecter ms_handle_connect 0x7f0fec13f650 2024-03-29T11:25:39.211 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.221+0000 7f0fe97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:39.211 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.221+0000 7f0fe97fa640 1 -- 172.21.15.114:0/1495865435 <== mon.2 v1:172.21.15.114:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2246319804 0 0) 0x7f0fec0ac7b0 con 0x7f0fec1457f0 2024-03-29T11:25:39.211 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.221+0000 7f0fe97fa640 1 -- 172.21.15.114:0/1495865435 >> v1:172.21.15.176:6789/0 conn(0x7f0fec0a9d70 legacy=0x7f0fec137210 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:25:39.211 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.223+0000 7f0fe97fa640 1 -- 172.21.15.114:0/1495865435 >> v1:172.21.15.114:6789/0 conn(0x7f0fec13f650 legacy=0x7f0fec141ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:39.211 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.223+0000 7f0fe97fa640 10 monclient: _finish_hunting 0 2024-03-29T11:25:39.211 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.223+0000 7f0fe97fa640 1 monclient: found mon.c 2024-03-29T11:25:39.211 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.223+0000 7f0fe97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:39.211 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.223+0000 7f0fe97fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:25:39.211 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.223+0000 7f0fe97fa640 1 -- 172.21.15.114:0/1495865435 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fec0ab620 con 0x7f0fec1457f0 2024-03-29T11:25:39.211 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.223+0000 7f0fe97fa640 10 monclient: _finish_auth 0 2024-03-29T11:25:39.211 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.223+0000 7f0fe97fa640 10 monclient: _check_auth_tickets 2024-03-29T11:25:39.211 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.223+0000 7f0fe97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:39.211 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.223+0000 7f0ff2af1640 5 monclient: authenticate success, global_id 4418 2024-03-29T11:25:39.211 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.223+0000 7f0ff2af1640 10 monclient: _renew_subs 2024-03-29T11:25:39.211 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.223+0000 7f0ff2af1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:25:39.212 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.223+0000 7f0ff2af1640 1 -- 172.21.15.114:0/1495865435 --> v1:172.21.15.114:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fec08bcc0 con 0x7f0fec1457f0 2024-03-29T11:25:39.212 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.223+0000 7f0ff2af1640 10 client.4418.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:25:39.212 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.223+0000 7f0ff2af1640 10 monclient: _renew_subs 2024-03-29T11:25:39.212 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.223+0000 7f0ff2af1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:25:39.212 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.223+0000 7f0ff2af1640 1 -- 172.21.15.114:0/1495865435 --> v1:172.21.15.114:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fec0b1490 con 0x7f0fec1457f0 2024-03-29T11:25:39.212 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.223+0000 7f0ff2af1640 1 librados: init done 2024-03-29T11:25:39.212 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.224+0000 7f0fe97fa640 1 -- 172.21.15.114:0/1495865435 <== mon.2 v1:172.21.15.114:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0fe4001aa0 con 0x7f0fec1457f0 2024-03-29T11:25:39.212 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.224+0000 7f0fe97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:39.212 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.224+0000 7f0fe97fa640 1 -- 172.21.15.114:0/1495865435 <== mon.2 v1:172.21.15.114:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f0fe4005210 con 0x7f0fec1457f0 2024-03-29T11:25:39.212 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.224+0000 7f0feaffd640 4 set_mon_vals no callback set 2024-03-29T11:25:39.214 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.226+0000 7f0fe97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:39.214 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.226+0000 7f0fe97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:25:39.215 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.226+0000 7f0fe97fa640 10 monclient: dump: 2024-03-29T11:25:39.215 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:25:39.215 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:25:39.215 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:25:39.215 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:25:39.215 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:39.215 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:25:39.215 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:25:39.215 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:25:39.215 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:25:39.215 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:25:39.215 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.226+0000 7f0fe97fa640 1 -- 172.21.15.114:0/1495865435 <== mon.2 v1:172.21.15.114:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1594292461 0 0) 0x7f0fe401b1e0 con 0x7f0fec1457f0 2024-03-29T11:25:39.216 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.227+0000 7f0fe97fa640 1 -- 172.21.15.114:0/1495865435 <== mon.2 v1:172.21.15.114:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3238987313 0 0) 0x7f0fe4085000 con 0x7f0fec1457f0 2024-03-29T11:25:39.216 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.227+0000 7f0ff088d640 10 monclient: get_auth_request con 0x7f0fd806dc40 auth_method 0 2024-03-29T11:25:39.216 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.226+0000 7f0ff2af1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:39.216 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.227+0000 7f0ff2af1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:39.216 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.227+0000 7f0ff2af1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:25:39.216 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.227+0000 7f0ff2af1640 1 -- 172.21.15.114:0/1495865435 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fcc005080 con 0x7f0fec1457f0 2024-03-29T11:25:39.216 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.228+0000 7f0fe97fa640 10 client.4418.objecter ms_dispatch 0x7f0fec000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:25:39.216 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.228+0000 7f0fe97fa640 3 client.4418.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:25:39.216 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.228+0000 7f0fe97fa640 3 client.4418.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:25:39.216 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.228+0000 7f0fe97fa640 20 client.4418.objecter dump_active .. 0 homeless 2024-03-29T11:25:39.216 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.228+0000 7f0fe97fa640 10 client.4418.objecter ms_handle_connect 0x7f0fd806dc40 2024-03-29T11:25:39.223 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.239+0000 7f0fe97fa640 1 -- 172.21.15.114:0/1495865435 <== mon.2 v1:172.21.15.114:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0fe4050710 con 0x7f0fec1457f0 2024-03-29T11:25:39.223 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.239+0000 7f0fe97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:39.223 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.239+0000 7f0fe97fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:25:39.554 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.569+0000 7f0ff2af1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:39.554 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.569+0000 7f0ff2af1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:39.554 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.569+0000 7f0ff2af1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:25:39.554 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.569+0000 7f0ff2af1640 1 -- 172.21.15.114:0/1495865435 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0fcc005330 con 0x7f0fec1457f0 2024-03-29T11:25:39.554 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.570+0000 7f0fe97fa640 1 -- 172.21.15.114:0/1495865435 <== mon.2 v1:172.21.15.114:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2858224613) 0x7f0fe4052810 con 0x7f0fec1457f0 2024-03-29T11:25:39.554 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.570+0000 7f0fe97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:39.554 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.570+0000 7f0fe97fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:25:39.555 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T11:25:39.555 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"b0de2b9b-c45a-44c7-b6ee-e83b39a1e108","created":"2024-03-29T11:20:49.420660+0000","modified":"2024-03-29T11:20:51.700167+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:51.700167+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a64785d3-6b11-499a-b706-cc827ab96a74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5882dadb-e7a8-4604-ba9a-418013c106c8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d7e8c409-bc66-4e4a-bbdb-420bad31388e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0682989-3595-4cd6-90a5-ab9111c278e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"67068120-e3ff-4817-bd06-44feb7479a35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09e8b-8a85-4997-bdea-75d754e6ee67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2f0d9ead-f47f-4cb3-92a4-0a5ad23cd831","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fdb6a25f-6bd8-4c65-9b9b-b460ba7673a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:25:39.558 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.573+0000 7f0ff2af1640 10 librados: watch_flush enter 2024-03-29T11:25:39.558 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.573+0000 7f0ff2af1640 10 librados: watch_flush exit 2024-03-29T11:25:39.558 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.573+0000 7f0ff2af1640 20 client.4418.objecter shutdown clearing up homeless session... 2024-03-29T11:25:39.558 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.573+0000 7f0ff2af1640 10 client.4418.objecter successfully canceled tick 2024-03-29T11:25:39.559 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.574+0000 7f0ff2af1640 1 -- 172.21.15.114:0/1495865435 >> v1:172.21.15.114:6816/4038755055 conn(0x7f0fd806dc40 legacy=0x7f0fd8070100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:39.559 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.574+0000 7f0ff2af1640 10 monclient: shutdown 2024-03-29T11:25:39.559 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.574+0000 7f0ff2af1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:39.559 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.574+0000 7f0ff2af1640 1 -- 172.21.15.114:0/1495865435 >> v1:172.21.15.114:6790/0 conn(0x7f0fec1457f0 legacy=0x7f0fec1f3280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:39.559 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.574+0000 7f0ff2af1640 1 -- 172.21.15.114:0/1495865435 shutdown_connections 2024-03-29T11:25:39.559 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.574+0000 7f0ff2af1640 1 -- 172.21.15.114:0/1495865435 >> 172.21.15.114:0/1495865435 conn(0x7f0fec066a00 msgr2=0x7f0fec13aaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:39.559 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.574+0000 7f0ff2af1640 1 -- 172.21.15.114:0/1495865435 shutdown_connections 2024-03-29T11:25:39.559 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.574+0000 7f0ff2af1640 1 -- 172.21.15.114:0/1495865435 wait complete. 2024-03-29T11:25:39.559 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:39.575+0000 7f0ff2af1640 1 librados: shutdown 2024-03-29T11:25:39.576 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:25:46.383 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:25:46.531 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.546+0000 7fe166ec0640 10 monclient: get_monmap_and_config 2024-03-29T11:25:46.531 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.546+0000 7fe166ec0640 10 monclient: build_initial_monmap 2024-03-29T11:25:46.531 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.546+0000 7fe166ec0640 10 monclient: monmap: 2024-03-29T11:25:46.532 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:25:46.532 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:25:46.532 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:25:46.547577+0000 2024-03-29T11:25:46.532 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:25:46.547577+0000 2024-03-29T11:25:46.532 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:46.532 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:25:46.532 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:25:46.532 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T11:25:46.532 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T11:25:46.532 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:25:46.533 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.547+0000 7fe166ec0640 1 Processor -- start 2024-03-29T11:25:46.533 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.547+0000 7fe166ec0640 1 -- start start 2024-03-29T11:25:46.533 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.547+0000 7fe166ec0640 10 monclient: init 2024-03-29T11:25:46.534 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.548+0000 7fe166ec0640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:46.535 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.548+0000 7fe166ec0640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:25:46.535 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.548+0000 7fe166ec0640 10 monclient(hunting): picked mon.noname-c con 0x7fe1601457f0 addr v1:172.21.15.176:6789/0 2024-03-29T11:25:46.535 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.548+0000 7fe166ec0640 10 monclient(hunting): picked mon.noname-a con 0x7fe16013f650 addr v1:172.21.15.114:6789/0 2024-03-29T11:25:46.535 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.548+0000 7fe166ec0640 10 monclient(hunting): picked mon.noname-b con 0x7fe1600a9d70 addr v1:172.21.15.114:6790/0 2024-03-29T11:25:46.535 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.548+0000 7fe166ec0640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1600aaef0 con 0x7fe16013f650 2024-03-29T11:25:46.535 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.548+0000 7fe166ec0640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe160145bd0 con 0x7fe1601457f0 2024-03-29T11:25:46.535 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.548+0000 7fe166ec0640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe160145eb0 con 0x7fe1600a9d70 2024-03-29T11:25:46.535 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.548+0000 7fe166ec0640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:46.535 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.548+0000 7fe166ec0640 10 monclient(hunting): authenticate will time out at 1216.730833s 2024-03-29T11:25:46.536 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.551+0000 7fe16545d640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7fe1600a9d70 0x7fe16014d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:33176/0 (socket says 172.21.15.114:33176) 2024-03-29T11:25:46.536 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.551+0000 7fe16545d640 1 -- 172.21.15.114:0/2836209442 learned_addr learned my addr 172.21.15.114:0/2836209442 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:25:46.539 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.553+0000 7fe15f7fe640 1 -- 172.21.15.114:0/2836209442 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1097439724 0 0) 0x7fe160145eb0 con 0x7fe1600a9d70 2024-03-29T11:25:46.539 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.553+0000 7fe15f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:25:46.539 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.553+0000 7fe15f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:46.539 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.553+0000 7fe15f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:46.542 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.553+0000 7fe15f7fe640 10 monclient(hunting): my global_id is 4421 2024-03-29T11:25:46.542 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.553+0000 7fe15f7fe640 1 -- 172.21.15.114:0/2836209442 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe148003520 con 0x7fe1600a9d70 2024-03-29T11:25:46.542 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.554+0000 7fe15f7fe640 1 -- 172.21.15.114:0/2836209442 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2450769209 0 0) 0x7fe148003520 con 0x7fe1600a9d70 2024-03-29T11:25:46.542 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.554+0000 7fe15f7fe640 1 -- 172.21.15.114:0/2836209442 >> v1:172.21.15.176:6789/0 conn(0x7fe1601457f0 legacy=0x7fe16013ef40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T11:25:46.542 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.554+0000 7fe15f7fe640 1 -- 172.21.15.114:0/2836209442 >> v1:172.21.15.114:6789/0 conn(0x7fe16013f650 legacy=0x7fe1600b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:25:46.542 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.554+0000 7fe15f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:25:46.542 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.554+0000 7fe15f7fe640 1 monclient: found mon.noname-b 2024-03-29T11:25:46.542 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.554+0000 7fe15f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:46.542 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.554+0000 7fe15f7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.114:6790/0 2024-03-29T11:25:46.543 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.554+0000 7fe15f7fe640 1 -- 172.21.15.114:0/2836209442 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe160146340 con 0x7fe1600a9d70 2024-03-29T11:25:46.543 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.554+0000 7fe15f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:25:46.543 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.554+0000 7fe15f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:25:46.543 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.554+0000 7fe15f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:46.543 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.554+0000 7fe15f7fe640 1 -- 172.21.15.114:0/2836209442 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fe150001690 con 0x7fe1600a9d70 2024-03-29T11:25:46.543 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.554+0000 7fe15f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:46.543 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.554+0000 7fe15f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:25:46.543 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.554+0000 7fe15f7fe640 10 monclient: dump: 2024-03-29T11:25:46.543 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:25:46.543 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:25:46.543 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:25:46.543 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:25:46.543 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:46.543 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:25:46.544 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:25:46.544 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:25:46.544 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:25:46.544 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:25:46.544 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.554+0000 7fe166ec0640 5 monclient: authenticate success, global_id 4421 2024-03-29T11:25:46.544 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.554+0000 7fe166ec0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:46.544 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.554+0000 7fe15f7fe640 1 -- 172.21.15.114:0/2836209442 <== mon.2 v1:172.21.15.114:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe150001860 con 0x7fe1600a9d70 2024-03-29T11:25:46.544 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.554+0000 7fe15f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:46.544 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.554+0000 7fe166ec0640 10 monclient: get_monmap_and_config success 2024-03-29T11:25:46.544 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.554+0000 7fe166ec0640 4 set_mon_vals no callback set 2024-03-29T11:25:46.544 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.555+0000 7fe166ec0640 10 monclient: shutdown 2024-03-29T11:25:46.544 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.555+0000 7fe166ec0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:46.544 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.555+0000 7fe166ec0640 1 -- 172.21.15.114:0/2836209442 >> v1:172.21.15.114:6790/0 conn(0x7fe1600a9d70 legacy=0x7fe16014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:46.544 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.555+0000 7fe166ec0640 1 -- 172.21.15.114:0/2836209442 shutdown_connections 2024-03-29T11:25:46.545 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.555+0000 7fe166ec0640 1 -- 172.21.15.114:0/2836209442 >> 172.21.15.114:0/2836209442 conn(0x7fe160066a00 msgr2=0x7fe160066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:46.545 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.555+0000 7fe166ec0640 1 -- 172.21.15.114:0/2836209442 shutdown_connections 2024-03-29T11:25:46.545 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.555+0000 7fe166ec0640 1 -- 172.21.15.114:0/2836209442 wait complete. 2024-03-29T11:25:46.551 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.557+0000 7fe166ec0640 10 monclient: build_initial_monmap 2024-03-29T11:25:46.551 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.557+0000 7fe166ec0640 10 monclient: monmap: 2024-03-29T11:25:46.551 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:25:46.551 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:25:46.551 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T11:25:46.552 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T11:25:46.552 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:46.552 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:25:46.552 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:25:46.552 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T11:25:46.552 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T11:25:46.552 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:25:46.552 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.557+0000 7fe166ec0640 1 librados: starting msgr at 2024-03-29T11:25:46.552 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.557+0000 7fe166ec0640 1 librados: starting objecter 2024-03-29T11:25:46.552 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.557+0000 7fe166ec0640 1 Processor -- start 2024-03-29T11:25:46.552 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.557+0000 7fe166ec0640 1 -- start start 2024-03-29T11:25:46.552 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.557+0000 7fe166ec0640 1 librados: setting wanted keys 2024-03-29T11:25:46.552 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.557+0000 7fe166ec0640 1 librados: calling monclient init 2024-03-29T11:25:46.552 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.557+0000 7fe166ec0640 10 monclient: init 2024-03-29T11:25:46.553 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.558+0000 7fe166ec0640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:46.553 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.558+0000 7fe166ec0640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:25:46.553 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.558+0000 7fe166ec0640 10 monclient(hunting): picked mon.noname-b con 0x7fe1600a9d70 addr v1:172.21.15.176:6789/0 2024-03-29T11:25:46.553 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.558+0000 7fe166ec0640 10 monclient(hunting): picked mon.noname-c con 0x7fe16013f650 addr v1:172.21.15.114:6790/0 2024-03-29T11:25:46.553 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.558+0000 7fe166ec0640 10 monclient(hunting): picked mon.noname-a con 0x7fe1601457f0 addr v1:172.21.15.114:6789/0 2024-03-29T11:25:46.553 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.558+0000 7fe166ec0640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe160132740 con 0x7fe1601457f0 2024-03-29T11:25:46.553 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.558+0000 7fe166ec0640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe160138ca0 con 0x7fe1600a9d70 2024-03-29T11:25:46.553 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.558+0000 7fe166ec0640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe160138f80 con 0x7fe16013f650 2024-03-29T11:25:46.553 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.558+0000 7fe166ec0640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:46.553 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.558+0000 7fe166ec0640 10 monclient(hunting): authenticate will time out at 1216.740963s 2024-03-29T11:25:46.553 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.560+0000 7fe15ffff640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7fe16013f650 0x7fe160131920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:33978/0 (socket says 172.21.15.114:33978) 2024-03-29T11:25:46.553 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.560+0000 7fe15ffff640 1 -- 172.21.15.114:0/3557373019 learned_addr learned my addr 172.21.15.114:0/3557373019 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:25:46.553 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.562+0000 7fe15d7fa640 10 client.?.objecter ms_handle_connect 0x7fe16013f650 2024-03-29T11:25:46.553 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.562+0000 7fe15d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:46.554 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.562+0000 7fe15d7fa640 1 -- 172.21.15.114:0/3557373019 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1341803456 0 0) 0x7fe160138f80 con 0x7fe16013f650 2024-03-29T11:25:46.554 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.562+0000 7fe15d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:25:46.554 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.562+0000 7fe15d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:46.554 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.562+0000 7fe15d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:46.554 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.562+0000 7fe15d7fa640 10 monclient(hunting): my global_id is 4424 2024-03-29T11:25:46.554 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.562+0000 7fe15d7fa640 1 -- 172.21.15.114:0/3557373019 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe14c004eb0 con 0x7fe16013f650 2024-03-29T11:25:46.554 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.563+0000 7fe15d7fa640 10 client.?.objecter ms_handle_connect 0x7fe1601457f0 2024-03-29T11:25:46.554 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.563+0000 7fe15d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:46.554 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.563+0000 7fe15d7fa640 1 -- 172.21.15.114:0/3557373019 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 20385453 0 0) 0x7fe14c004eb0 con 0x7fe16013f650 2024-03-29T11:25:46.554 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.563+0000 7fe15d7fa640 1 -- 172.21.15.114:0/3557373019 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe160138f80 con 0x7fe16013f650 2024-03-29T11:25:46.554 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.563+0000 7fe15d7fa640 1 -- 172.21.15.114:0/3557373019 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fe154001960 con 0x7fe16013f650 2024-03-29T11:25:46.554 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.563+0000 7fe15d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:46.554 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.563+0000 7fe15d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:25:46.555 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.563+0000 7fe15d7fa640 10 monclient(hunting): dump: 2024-03-29T11:25:46.555 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:25:46.555 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:25:46.555 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:25:46.555 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:25:46.555 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:46.555 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:25:46.555 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:25:46.555 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:25:46.555 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:25:46.555 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:25:46.555 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.563+0000 7fe15d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:25:46.555 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.564+0000 7fe15d7fa640 1 -- 172.21.15.114:0/3557373019 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1301198153 0 0) 0x7fe160132740 con 0x7fe1601457f0 2024-03-29T11:25:46.556 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.564+0000 7fe15d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:25:46.556 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.564+0000 7fe15d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:46.556 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.564+0000 7fe15d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:46.556 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.564+0000 7fe15d7fa640 10 monclient(hunting): my global_id is 4398 2024-03-29T11:25:46.556 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.564+0000 7fe15d7fa640 1 -- 172.21.15.114:0/3557373019 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe14c004eb0 con 0x7fe1601457f0 2024-03-29T11:25:46.556 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.564+0000 7fe15d7fa640 1 -- 172.21.15.114:0/3557373019 <== mon.2 v1:172.21.15.114:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 352899053 0 0) 0x7fe160138f80 con 0x7fe16013f650 2024-03-29T11:25:46.556 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.564+0000 7fe15d7fa640 1 -- 172.21.15.114:0/3557373019 >> v1:172.21.15.176:6789/0 conn(0x7fe1600a9d70 legacy=0x7fe160139680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:46.556 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.564+0000 7fe15d7fa640 1 -- 172.21.15.114:0/3557373019 >> v1:172.21.15.114:6789/0 conn(0x7fe1601457f0 legacy=0x7fe160132030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:46.556 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.564+0000 7fe15d7fa640 10 monclient: _finish_hunting 0 2024-03-29T11:25:46.556 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.564+0000 7fe15d7fa640 1 monclient: found mon.c 2024-03-29T11:25:46.556 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.564+0000 7fe15d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:46.556 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.565+0000 7fe15d7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:25:46.556 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.565+0000 7fe15d7fa640 1 -- 172.21.15.114:0/3557373019 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe160145010 con 0x7fe16013f650 2024-03-29T11:25:46.556 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.565+0000 7fe15d7fa640 10 monclient: _finish_auth 0 2024-03-29T11:25:46.557 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.565+0000 7fe15d7fa640 10 monclient: _check_auth_tickets 2024-03-29T11:25:46.557 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.565+0000 7fe15d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:46.557 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.565+0000 7fe166ec0640 5 monclient: authenticate success, global_id 4424 2024-03-29T11:25:46.557 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.565+0000 7fe166ec0640 10 monclient: _renew_subs 2024-03-29T11:25:46.557 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.565+0000 7fe166ec0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:25:46.557 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.565+0000 7fe166ec0640 1 -- 172.21.15.114:0/3557373019 --> v1:172.21.15.114:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe160138640 con 0x7fe16013f650 2024-03-29T11:25:46.557 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.565+0000 7fe166ec0640 10 client.4424.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:25:46.557 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.565+0000 7fe166ec0640 10 monclient: _renew_subs 2024-03-29T11:25:46.557 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.565+0000 7fe166ec0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:25:46.557 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.565+0000 7fe166ec0640 1 -- 172.21.15.114:0/3557373019 --> v1:172.21.15.114:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe16013dd90 con 0x7fe16013f650 2024-03-29T11:25:46.557 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.565+0000 7fe166ec0640 1 librados: init done 2024-03-29T11:25:46.557 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.565+0000 7fe15d7fa640 10 client.4424.objecter ms_handle_connect 0x7fe1600a9d70 2024-03-29T11:25:46.557 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.566+0000 7fe15d7fa640 10 client.4424.objecter resend_mon_ops 2024-03-29T11:25:46.557 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.566+0000 7fe15d7fa640 1 -- 172.21.15.114:0/3557373019 <== mon.2 v1:172.21.15.114:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe154001d70 con 0x7fe16013f650 2024-03-29T11:25:46.558 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.566+0000 7fe15d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:46.559 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.566+0000 7fe15e7fc640 4 set_mon_vals no callback set 2024-03-29T11:25:46.559 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.567+0000 7fe15d7fa640 1 -- 172.21.15.114:0/3557373019 <== mon.2 v1:172.21.15.114:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fe154001b80 con 0x7fe16013f650 2024-03-29T11:25:46.559 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.567+0000 7fe15d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:46.560 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.567+0000 7fe15d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:25:46.560 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.567+0000 7fe15d7fa640 10 monclient: dump: 2024-03-29T11:25:46.560 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:25:46.560 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:25:46.560 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:25:46.560 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:25:46.560 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:46.560 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:25:46.560 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:25:46.560 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:25:46.560 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:25:46.560 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:25:46.560 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.568+0000 7fe15d7fa640 1 -- 172.21.15.114:0/3557373019 <== mon.2 v1:172.21.15.114:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1594292461 0 0) 0x7fe154003b40 con 0x7fe16013f650 2024-03-29T11:25:46.560 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.568+0000 7fe15d7fa640 1 -- 172.21.15.114:0/3557373019 <== mon.2 v1:172.21.15.114:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3238987313 0 0) 0x7fe154084b00 con 0x7fe16013f650 2024-03-29T11:25:46.561 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.569+0000 7fe15d7fa640 10 client.4424.objecter ms_dispatch 0x7fe160000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:25:46.561 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.569+0000 7fe15d7fa640 3 client.4424.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:25:46.561 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.569+0000 7fe15d7fa640 3 client.4424.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:25:46.561 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.569+0000 7fe15d7fa640 20 client.4424.objecter dump_active .. 0 homeless 2024-03-29T11:25:46.561 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.569+0000 7fe164c5c640 10 monclient: get_auth_request con 0x7fe14c06dcc0 auth_method 0 2024-03-29T11:25:46.561 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.569+0000 7fe166ec0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:46.561 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.570+0000 7fe166ec0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:46.561 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.570+0000 7fe166ec0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:25:46.561 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.570+0000 7fe166ec0640 1 -- 172.21.15.114:0/3557373019 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe140005080 con 0x7fe16013f650 2024-03-29T11:25:46.561 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.570+0000 7fe15d7fa640 10 client.4424.objecter ms_handle_connect 0x7fe14c06dcc0 2024-03-29T11:25:46.564 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.579+0000 7fe15d7fa640 1 -- 172.21.15.114:0/3557373019 <== mon.2 v1:172.21.15.114:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe1540503e0 con 0x7fe16013f650 2024-03-29T11:25:46.564 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.579+0000 7fe15d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:46.565 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.579+0000 7fe15d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:25:46.892 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.907+0000 7fe166ec0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:46.892 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.907+0000 7fe166ec0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:46.892 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.907+0000 7fe166ec0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:25:46.892 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.907+0000 7fe166ec0640 1 -- 172.21.15.114:0/3557373019 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe140005330 con 0x7fe16013f650 2024-03-29T11:25:46.894 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.909+0000 7fe15d7fa640 1 -- 172.21.15.114:0/3557373019 <== mon.2 v1:172.21.15.114:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2858224613) 0x7fe154052310 con 0x7fe16013f650 2024-03-29T11:25:46.894 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.909+0000 7fe15d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:46.894 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.909+0000 7fe15d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:25:46.894 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T11:25:46.894 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"b0de2b9b-c45a-44c7-b6ee-e83b39a1e108","created":"2024-03-29T11:20:49.420660+0000","modified":"2024-03-29T11:20:51.700167+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:51.700167+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a64785d3-6b11-499a-b706-cc827ab96a74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5882dadb-e7a8-4604-ba9a-418013c106c8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d7e8c409-bc66-4e4a-bbdb-420bad31388e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0682989-3595-4cd6-90a5-ab9111c278e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"67068120-e3ff-4817-bd06-44feb7479a35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09e8b-8a85-4997-bdea-75d754e6ee67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2f0d9ead-f47f-4cb3-92a4-0a5ad23cd831","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fdb6a25f-6bd8-4c65-9b9b-b460ba7673a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:25:46.897 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.912+0000 7fe166ec0640 10 librados: watch_flush enter 2024-03-29T11:25:46.897 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.913+0000 7fe166ec0640 10 librados: watch_flush exit 2024-03-29T11:25:46.898 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.913+0000 7fe166ec0640 20 client.4424.objecter shutdown clearing up homeless session... 2024-03-29T11:25:46.898 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.913+0000 7fe166ec0640 10 client.4424.objecter successfully canceled tick 2024-03-29T11:25:46.898 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.913+0000 7fe166ec0640 1 -- 172.21.15.114:0/3557373019 >> v1:172.21.15.114:6816/4038755055 conn(0x7fe14c06dcc0 legacy=0x7fe14c070180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:46.898 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.913+0000 7fe166ec0640 10 monclient: shutdown 2024-03-29T11:25:46.898 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.913+0000 7fe166ec0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:46.898 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.913+0000 7fe166ec0640 1 -- 172.21.15.114:0/3557373019 >> v1:172.21.15.114:6790/0 conn(0x7fe16013f650 legacy=0x7fe160131920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:46.901 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.916+0000 7fe166ec0640 1 -- 172.21.15.114:0/3557373019 shutdown_connections 2024-03-29T11:25:46.901 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.916+0000 7fe166ec0640 1 -- 172.21.15.114:0/3557373019 >> 172.21.15.114:0/3557373019 conn(0x7fe160066a00 msgr2=0x7fe16013a470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:46.901 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.916+0000 7fe166ec0640 1 -- 172.21.15.114:0/3557373019 shutdown_connections 2024-03-29T11:25:46.901 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.916+0000 7fe166ec0640 1 -- 172.21.15.114:0/3557373019 wait complete. 2024-03-29T11:25:46.901 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:46.917+0000 7fe166ec0640 1 librados: shutdown 2024-03-29T11:25:46.918 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:25:53.725 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:25:53.877 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.891+0000 7f34b31b6640 10 monclient: get_monmap_and_config 2024-03-29T11:25:53.877 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.891+0000 7f34b31b6640 10 monclient: build_initial_monmap 2024-03-29T11:25:53.877 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.891+0000 7f34b31b6640 10 monclient: monmap: 2024-03-29T11:25:53.877 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:25:53.877 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:25:53.878 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:25:53.892920+0000 2024-03-29T11:25:53.878 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:25:53.892920+0000 2024-03-29T11:25:53.878 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:53.878 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:25:53.878 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:25:53.878 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T11:25:53.878 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T11:25:53.878 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:25:53.879 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.894+0000 7f34b31b6640 1 Processor -- start 2024-03-29T11:25:53.884 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.894+0000 7f34b31b6640 1 -- start start 2024-03-29T11:25:53.884 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.894+0000 7f34b31b6640 10 monclient: init 2024-03-29T11:25:53.884 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.895+0000 7f34b31b6640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:53.884 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.895+0000 7f34b31b6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:25:53.884 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.895+0000 7f34b31b6640 10 monclient(hunting): picked mon.noname-b con 0x7f34a4006a90 addr v1:172.21.15.114:6790/0 2024-03-29T11:25:53.884 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.895+0000 7f34b31b6640 10 monclient(hunting): picked mon.noname-c con 0x7f34a402b120 addr v1:172.21.15.176:6789/0 2024-03-29T11:25:53.884 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.898+0000 7f34b1753640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f34a4029c50 0x7f34a40d74c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:60271/0 (socket says 172.21.15.114:60271) 2024-03-29T11:25:53.884 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.899+0000 7f34b1753640 1 -- 172.21.15.114:0/3590364212 learned_addr learned my addr 172.21.15.114:0/3590364212 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:25:53.885 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.899+0000 7f34b31b6640 10 monclient(hunting): picked mon.noname-a con 0x7f34a4029c50 addr v1:172.21.15.114:6789/0 2024-03-29T11:25:53.886 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.899+0000 7f34b31b6640 1 -- 172.21.15.114:0/3590364212 --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34a402a290 con 0x7f34a4029c50 2024-03-29T11:25:53.887 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.899+0000 7f34b31b6640 1 -- 172.21.15.114:0/3590364212 --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34a402c460 con 0x7f34a402b120 2024-03-29T11:25:53.887 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.899+0000 7f34b31b6640 1 -- 172.21.15.114:0/3590364212 --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34a40c9460 con 0x7f34a4006a90 2024-03-29T11:25:53.887 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.899+0000 7f34b31b6640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:53.887 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.899+0000 7f34b31b6640 10 monclient(hunting): authenticate will time out at 1224.081511s 2024-03-29T11:25:53.887 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.899+0000 7f34ab7fe640 1 -- 172.21.15.114:0/3590364212 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4278316790 0 0) 0x7f34a402a290 con 0x7f34a4029c50 2024-03-29T11:25:53.887 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.899+0000 7f34ab7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:25:53.887 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.899+0000 7f34ab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:53.887 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.899+0000 7f34ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:53.887 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.899+0000 7f34ab7fe640 10 monclient(hunting): my global_id is 4401 2024-03-29T11:25:53.887 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.899+0000 7f34ab7fe640 1 -- 172.21.15.114:0/3590364212 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f349c003520 con 0x7f34a4029c50 2024-03-29T11:25:53.887 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.900+0000 7f34ab7fe640 1 -- 172.21.15.114:0/3590364212 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4257228520 0 0) 0x7f349c003520 con 0x7f34a4029c50 2024-03-29T11:25:53.887 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.900+0000 7f34ab7fe640 1 -- 172.21.15.114:0/3590364212 >> v1:172.21.15.114:6790/0 conn(0x7f34a4006a90 legacy=0x7f34a40c8d50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:25:53.887 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.901+0000 7f34ab7fe640 1 -- 172.21.15.114:0/3590364212 >> v1:172.21.15.176:6789/0 conn(0x7f34a402b120 legacy=0x7f34a40cf120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:53.887 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.901+0000 7f34ab7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:25:53.888 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.901+0000 7f34ab7fe640 1 monclient: found mon.noname-a 2024-03-29T11:25:53.888 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.901+0000 7f34ab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:53.888 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.901+0000 7f34ab7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.114:6789/0 2024-03-29T11:25:53.888 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.901+0000 7f34ab7fe640 1 -- 172.21.15.114:0/3590364212 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34a40c95f0 con 0x7f34a4029c50 2024-03-29T11:25:53.888 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.901+0000 7f34ab7fe640 10 monclient: _finish_auth 0 2024-03-29T11:25:53.888 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.901+0000 7f34ab7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:25:53.889 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.901+0000 7f34ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:53.889 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.901+0000 7f34b31b6640 5 monclient: authenticate success, global_id 4401 2024-03-29T11:25:53.889 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.901+0000 7f34b31b6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:53.889 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.901+0000 7f34ab7fe640 1 -- 172.21.15.114:0/3590364212 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f34ac04f730 con 0x7f34a4029c50 2024-03-29T11:25:53.889 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.901+0000 7f34ab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:53.889 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.901+0000 7f34ab7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:25:53.889 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.901+0000 7f34ab7fe640 10 monclient: dump: 2024-03-29T11:25:53.889 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:25:53.889 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:25:53.889 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:25:53.889 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:25:53.889 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:53.889 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:25:53.889 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:25:53.890 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:25:53.890 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:25:53.890 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:25:53.890 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.901+0000 7f34b31b6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:53.890 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.903+0000 7f34ab7fe640 1 -- 172.21.15.114:0/3590364212 <== mon.0 v1:172.21.15.114:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f34ac0008d0 con 0x7f34a4029c50 2024-03-29T11:25:53.890 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.903+0000 7f34ab7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:53.891 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.903+0000 7f34b31b6640 10 monclient: get_monmap_and_config success 2024-03-29T11:25:53.891 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.903+0000 7f34b31b6640 4 set_mon_vals no callback set 2024-03-29T11:25:53.891 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.904+0000 7f34b31b6640 10 monclient: shutdown 2024-03-29T11:25:53.891 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.904+0000 7f34b31b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:53.891 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.904+0000 7f34b31b6640 1 -- 172.21.15.114:0/3590364212 >> v1:172.21.15.114:6789/0 conn(0x7f34a4029c50 legacy=0x7f34a40d74c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:53.891 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.904+0000 7f34b31b6640 1 -- 172.21.15.114:0/3590364212 shutdown_connections 2024-03-29T11:25:53.892 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.904+0000 7f34b31b6640 1 -- 172.21.15.114:0/3590364212 >> 172.21.15.114:0/3590364212 conn(0x7f34a4005910 msgr2=0x7f34a4005d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:53.892 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.905+0000 7f34b31b6640 1 -- 172.21.15.114:0/3590364212 shutdown_connections 2024-03-29T11:25:53.892 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.905+0000 7f34b31b6640 1 -- 172.21.15.114:0/3590364212 wait complete. 2024-03-29T11:25:53.892 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.906+0000 7f34b31b6640 10 monclient: build_initial_monmap 2024-03-29T11:25:53.892 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.906+0000 7f34b31b6640 10 monclient: monmap: 2024-03-29T11:25:53.892 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:25:53.892 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:25:53.892 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T11:25:53.892 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T11:25:53.892 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:53.892 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:25:53.893 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:25:53.893 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T11:25:53.893 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T11:25:53.893 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:25:53.893 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.906+0000 7f34b31b6640 1 librados: starting msgr at 2024-03-29T11:25:53.893 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.906+0000 7f34b31b6640 1 librados: starting objecter 2024-03-29T11:25:53.893 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.906+0000 7f34b31b6640 1 Processor -- start 2024-03-29T11:25:53.893 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.906+0000 7f34b31b6640 1 -- start start 2024-03-29T11:25:53.893 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.906+0000 7f34b31b6640 1 librados: setting wanted keys 2024-03-29T11:25:53.893 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.906+0000 7f34b31b6640 1 librados: calling monclient init 2024-03-29T11:25:53.893 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.906+0000 7f34b31b6640 10 monclient: init 2024-03-29T11:25:53.895 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.907+0000 7f34b31b6640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:53.895 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.907+0000 7f34b31b6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:25:53.895 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.907+0000 7f34b31b6640 10 monclient(hunting): picked mon.noname-c con 0x7f34a4006a90 addr v1:172.21.15.114:6790/0 2024-03-29T11:25:53.895 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.907+0000 7f34b31b6640 10 monclient(hunting): picked mon.noname-b con 0x7f34a416d220 addr v1:172.21.15.176:6789/0 2024-03-29T11:25:53.895 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.907+0000 7f34b31b6640 10 monclient(hunting): picked mon.noname-a con 0x7f34a40cc2e0 addr v1:172.21.15.114:6789/0 2024-03-29T11:25:53.895 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.907+0000 7f34b31b6640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34a40ce430 con 0x7f34a40cc2e0 2024-03-29T11:25:53.895 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.907+0000 7f34b31b6640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34a40ce660 con 0x7f34a416d220 2024-03-29T11:25:53.895 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.907+0000 7f34b31b6640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34a40cc800 con 0x7f34a4006a90 2024-03-29T11:25:53.895 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.907+0000 7f34b31b6640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:53.895 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.907+0000 7f34b31b6640 10 monclient(hunting): authenticate will time out at 1224.089955s 2024-03-29T11:25:53.897 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.910+0000 7f34b0f52640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f34a4006a90 0x7f34a40ccd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:60376/0 (socket says 172.21.15.114:60376) 2024-03-29T11:25:53.897 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.911+0000 7f34b0f52640 1 -- 172.21.15.114:0/1968253225 learned_addr learned my addr 172.21.15.114:0/1968253225 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:25:53.898 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.911+0000 7f34a97fa640 10 client.?.objecter ms_handle_connect 0x7f34a40cc2e0 2024-03-29T11:25:53.898 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.911+0000 7f34a97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:53.898 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.911+0000 7f34a97fa640 10 client.?.objecter ms_handle_connect 0x7f34a4006a90 2024-03-29T11:25:53.898 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.911+0000 7f34a97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:53.898 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.911+0000 7f34a97fa640 1 -- 172.21.15.114:0/1968253225 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3096223563 0 0) 0x7f34a40ce430 con 0x7f34a40cc2e0 2024-03-29T11:25:53.898 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.911+0000 7f34a97fa640 10 monclient(hunting): _negotiate 2024-03-29T11:25:53.898 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.911+0000 7f34a97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:53.898 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.911+0000 7f34a97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:53.898 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.911+0000 7f34a97fa640 10 monclient(hunting): my global_id is 4404 2024-03-29T11:25:53.898 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.911+0000 7f34a97fa640 1 -- 172.21.15.114:0/1968253225 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3490004eb0 con 0x7f34a40cc2e0 2024-03-29T11:25:53.898 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.911+0000 7f34a97fa640 1 -- 172.21.15.114:0/1968253225 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3664877580 0 0) 0x7f34a40cc800 con 0x7f34a4006a90 2024-03-29T11:25:53.898 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.912+0000 7f34a97fa640 10 monclient(hunting): _negotiate 2024-03-29T11:25:53.898 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.912+0000 7f34a97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:53.898 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.912+0000 7f34a97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:53.899 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.912+0000 7f34a97fa640 10 monclient(hunting): my global_id is 4427 2024-03-29T11:25:53.899 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.912+0000 7f34a97fa640 1 -- 172.21.15.114:0/1968253225 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34a40ce430 con 0x7f34a4006a90 2024-03-29T11:25:53.900 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.912+0000 7f34a97fa640 1 -- 172.21.15.114:0/1968253225 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4032350093 0 0) 0x7f3490004eb0 con 0x7f34a40cc2e0 2024-03-29T11:25:53.900 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.912+0000 7f34a97fa640 1 -- 172.21.15.114:0/1968253225 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f34a40cc800 con 0x7f34a40cc2e0 2024-03-29T11:25:53.900 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.912+0000 7f34a97fa640 1 -- 172.21.15.114:0/1968253225 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f34ac0008d0 con 0x7f34a40cc2e0 2024-03-29T11:25:53.900 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.912+0000 7f34a97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:53.900 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.912+0000 7f34a97fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:25:53.900 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.912+0000 7f34a97fa640 10 monclient(hunting): dump: 2024-03-29T11:25:53.900 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:25:53.901 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:25:53.901 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:25:53.901 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:25:53.901 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:53.901 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:25:53.901 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:25:53.901 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:25:53.901 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:25:53.901 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:25:53.901 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.912+0000 7f34a97fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:25:53.901 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.913+0000 7f34a97fa640 1 -- 172.21.15.114:0/1968253225 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2883738502 0 0) 0x7f34a40ce430 con 0x7f34a4006a90 2024-03-29T11:25:53.902 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.913+0000 7f34a97fa640 1 -- 172.21.15.114:0/1968253225 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3490004eb0 con 0x7f34a4006a90 2024-03-29T11:25:53.902 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.913+0000 7f34a97fa640 1 -- 172.21.15.114:0/1968253225 <== mon.0 v1:172.21.15.114:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 656522117 0 0) 0x7f34a40cc800 con 0x7f34a40cc2e0 2024-03-29T11:25:53.902 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.913+0000 7f34a97fa640 1 -- 172.21.15.114:0/1968253225 >> v1:172.21.15.114:6790/0 conn(0x7f34a4006a90 legacy=0x7f34a40ccd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:53.902 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.913+0000 7f34a97fa640 1 -- 172.21.15.114:0/1968253225 >> v1:172.21.15.176:6789/0 conn(0x7f34a416d220 legacy=0x7f34a41661c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:25:53.905 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.914+0000 7f34a97fa640 10 monclient: _finish_hunting 0 2024-03-29T11:25:53.905 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.914+0000 7f34a97fa640 1 monclient: found mon.a 2024-03-29T11:25:53.905 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.914+0000 7f34a97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:53.905 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.915+0000 7f34a97fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T11:25:53.905 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.915+0000 7f34a97fa640 1 -- 172.21.15.114:0/1968253225 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34a40c99a0 con 0x7f34a40cc2e0 2024-03-29T11:25:53.905 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.915+0000 7f34a97fa640 10 monclient: _finish_auth 0 2024-03-29T11:25:53.906 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.915+0000 7f34a97fa640 10 monclient: _check_auth_tickets 2024-03-29T11:25:53.906 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.915+0000 7f34a97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:53.906 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.915+0000 7f34b31b6640 5 monclient: authenticate success, global_id 4404 2024-03-29T11:25:53.906 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.915+0000 7f34b31b6640 10 monclient: _renew_subs 2024-03-29T11:25:53.906 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.915+0000 7f34b31b6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T11:25:53.906 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.915+0000 7f34b31b6640 1 -- 172.21.15.114:0/1968253225 --> v1:172.21.15.114:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34a40cd430 con 0x7f34a40cc2e0 2024-03-29T11:25:53.906 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.915+0000 7f34b31b6640 10 client.4404.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:25:53.906 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.915+0000 7f34b31b6640 10 monclient: _renew_subs 2024-03-29T11:25:53.906 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.915+0000 7f34b31b6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T11:25:53.906 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.915+0000 7f34b31b6640 1 -- 172.21.15.114:0/1968253225 --> v1:172.21.15.114:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f34a416c800 con 0x7f34a40cc2e0 2024-03-29T11:25:53.906 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.915+0000 7f34b31b6640 1 librados: init done 2024-03-29T11:25:53.906 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.916+0000 7f34a97fa640 1 -- 172.21.15.114:0/1968253225 <== mon.0 v1:172.21.15.114:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f34ac04fdd0 con 0x7f34a40cc2e0 2024-03-29T11:25:53.906 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.916+0000 7f34a97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:53.907 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.916+0000 7f34a97fa640 1 -- 172.21.15.114:0/1968253225 <== mon.0 v1:172.21.15.114:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f34ac04f730 con 0x7f34a40cc2e0 2024-03-29T11:25:53.907 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.916+0000 7f34a97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:53.907 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.916+0000 7f34a97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:25:53.907 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.916+0000 7f34a97fa640 10 monclient: dump: 2024-03-29T11:25:53.907 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:25:53.907 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:25:53.907 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:25:53.907 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:25:53.907 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:53.907 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:25:53.907 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:25:53.907 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:25:53.907 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:25:53.908 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:25:53.908 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.917+0000 7f34aaffd640 4 set_mon_vals no callback set 2024-03-29T11:25:53.908 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.917+0000 7f34b31b6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:53.908 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.918+0000 7f34b31b6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:53.908 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.918+0000 7f34b31b6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T11:25:53.908 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.918+0000 7f34b31b6640 1 -- 172.21.15.114:0/1968253225 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34a4004cf0 con 0x7f34a40cc2e0 2024-03-29T11:25:53.909 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.921+0000 7f34a97fa640 1 -- 172.21.15.114:0/1968253225 <== mon.0 v1:172.21.15.114:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1594292461 0 0) 0x7f34ac075710 con 0x7f34a40cc2e0 2024-03-29T11:25:53.909 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.921+0000 7f34a97fa640 1 -- 172.21.15.114:0/1968253225 <== mon.0 v1:172.21.15.114:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3238987313 0 0) 0x7f34ac0ff4f0 con 0x7f34a40cc2e0 2024-03-29T11:25:53.909 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.921+0000 7f34a97fa640 10 client.4404.objecter ms_dispatch 0x7f34ac000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:25:53.909 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.922+0000 7f34a97fa640 3 client.4404.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:25:53.909 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.922+0000 7f34a97fa640 3 client.4404.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:25:53.909 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.922+0000 7f34b0f52640 10 monclient: get_auth_request con 0x7f349006dc50 auth_method 0 2024-03-29T11:25:53.909 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.922+0000 7f34a97fa640 20 client.4404.objecter dump_active .. 0 homeless 2024-03-29T11:25:53.909 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.922+0000 7f34a97fa640 10 client.4404.objecter ms_handle_connect 0x7f349006dc50 2024-03-29T11:25:53.914 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.929+0000 7f34a97fa640 1 -- 172.21.15.114:0/1968253225 <== mon.0 v1:172.21.15.114:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f34ac0cadd0 con 0x7f34a40cc2e0 2024-03-29T11:25:53.914 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.929+0000 7f34a97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:53.914 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:53.929+0000 7f34a97fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:25:54.236 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:54.250+0000 7f34b31b6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:54.236 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:54.250+0000 7f34b31b6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:54.236 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:54.250+0000 7f34b31b6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T11:25:54.236 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:54.250+0000 7f34b31b6640 1 -- 172.21.15.114:0/1968253225 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f34a40cc990 con 0x7f34a40cc2e0 2024-03-29T11:25:54.239 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:54.254+0000 7f34a97fa640 1 -- 172.21.15.114:0/1968253225 <== mon.0 v1:172.21.15.114:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2858224613) 0x7f34ac0ccd00 con 0x7f34a40cc2e0 2024-03-29T11:25:54.239 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:54.254+0000 7f34a97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:54.239 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:54.254+0000 7f34a97fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:25:54.239 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T11:25:54.239 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"b0de2b9b-c45a-44c7-b6ee-e83b39a1e108","created":"2024-03-29T11:20:49.420660+0000","modified":"2024-03-29T11:20:51.700167+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:51.700167+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a64785d3-6b11-499a-b706-cc827ab96a74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5882dadb-e7a8-4604-ba9a-418013c106c8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d7e8c409-bc66-4e4a-bbdb-420bad31388e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0682989-3595-4cd6-90a5-ab9111c278e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"67068120-e3ff-4817-bd06-44feb7479a35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09e8b-8a85-4997-bdea-75d754e6ee67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2f0d9ead-f47f-4cb3-92a4-0a5ad23cd831","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fdb6a25f-6bd8-4c65-9b9b-b460ba7673a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:25:54.242 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:54.257+0000 7f34b31b6640 10 librados: watch_flush enter 2024-03-29T11:25:54.244 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:54.259+0000 7f34b31b6640 10 librados: watch_flush exit 2024-03-29T11:25:54.244 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:54.259+0000 7f34b31b6640 20 client.4404.objecter shutdown clearing up homeless session... 2024-03-29T11:25:54.244 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:54.259+0000 7f34b31b6640 10 client.4404.objecter successfully canceled tick 2024-03-29T11:25:54.244 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:54.259+0000 7f34b31b6640 1 -- 172.21.15.114:0/1968253225 >> v1:172.21.15.114:6816/4038755055 conn(0x7f349006dc50 legacy=0x7f3490070110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:54.244 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:54.259+0000 7f34b31b6640 10 monclient: shutdown 2024-03-29T11:25:54.244 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:54.259+0000 7f34b31b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:54.244 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:54.259+0000 7f34b31b6640 1 -- 172.21.15.114:0/1968253225 >> v1:172.21.15.114:6789/0 conn(0x7f34a40cc2e0 legacy=0x7f34a41748d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:54.245 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:54.260+0000 7f34b31b6640 1 -- 172.21.15.114:0/1968253225 shutdown_connections 2024-03-29T11:25:54.246 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:54.260+0000 7f34b31b6640 1 -- 172.21.15.114:0/1968253225 >> 172.21.15.114:0/1968253225 conn(0x7f34a4005910 msgr2=0x7f34a4037e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:54.246 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:54.260+0000 7f34b31b6640 1 -- 172.21.15.114:0/1968253225 shutdown_connections 2024-03-29T11:25:54.246 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:54.260+0000 7f34b31b6640 1 -- 172.21.15.114:0/1968253225 wait complete. 2024-03-29T11:25:54.246 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:25:54.260+0000 7f34b31b6640 1 librados: shutdown 2024-03-29T11:25:54.263 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:26:01.069 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:26:01.217 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.232+0000 7f7f40e51640 10 monclient: get_monmap_and_config 2024-03-29T11:26:01.218 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.232+0000 7f7f40e51640 10 monclient: build_initial_monmap 2024-03-29T11:26:01.218 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.232+0000 7f7f40e51640 10 monclient: monmap: 2024-03-29T11:26:01.218 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:26:01.218 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:26:01.218 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:26:01.233170+0000 2024-03-29T11:26:01.218 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:26:01.233170+0000 2024-03-29T11:26:01.218 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:01.218 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:26:01.218 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:26:01.218 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T11:26:01.218 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T11:26:01.219 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:26:01.219 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.233+0000 7f7f40e51640 1 Processor -- start 2024-03-29T11:26:01.220 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.233+0000 7f7f40e51640 1 -- start start 2024-03-29T11:26:01.220 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.233+0000 7f7f40e51640 10 monclient: init 2024-03-29T11:26:01.220 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.234+0000 7f7f40e51640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:01.220 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.234+0000 7f7f40e51640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:26:01.220 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.234+0000 7f7f40e51640 10 monclient(hunting): picked mon.noname-b con 0x7f7f34006890 addr v1:172.21.15.114:6790/0 2024-03-29T11:26:01.220 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.234+0000 7f7f40e51640 10 monclient(hunting): picked mon.noname-c con 0x7f7f34006c70 addr v1:172.21.15.176:6789/0 2024-03-29T11:26:01.220 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.234+0000 7f7f40e51640 10 monclient(hunting): picked mon.noname-a con 0x7f7f3402b4c0 addr v1:172.21.15.114:6789/0 2024-03-29T11:26:01.220 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.234+0000 7f7f40e51640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f3402c830 con 0x7f7f3402b4c0 2024-03-29T11:26:01.220 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.234+0000 7f7f40e51640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f3402afa0 con 0x7f7f34006c70 2024-03-29T11:26:01.220 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.234+0000 7f7f40e51640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f3402aba0 con 0x7f7f34006890 2024-03-29T11:26:01.220 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.234+0000 7f7f40e51640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:01.220 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.234+0000 7f7f40e51640 10 monclient(hunting): authenticate will time out at 1231.416985s 2024-03-29T11:26:01.223 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.238+0000 7f7f3bfff640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f7f3402b4c0 0x7f7f340cf0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:37159/0 (socket says 172.21.15.114:37159) 2024-03-29T11:26:01.223 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.238+0000 7f7f3bfff640 1 -- 172.21.15.114:0/2826605364 learned_addr learned my addr 172.21.15.114:0/2826605364 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:26:01.224 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.239+0000 7f7f3a7fc640 1 -- 172.21.15.114:0/2826605364 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3260795301 0 0) 0x7f7f3402c830 con 0x7f7f3402b4c0 2024-03-29T11:26:01.224 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.239+0000 7f7f3a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:26:01.224 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.239+0000 7f7f3a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:01.224 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.239+0000 7f7f3a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:01.224 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.239+0000 7f7f3a7fc640 10 monclient(hunting): my global_id is 4407 2024-03-29T11:26:01.224 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.239+0000 7f7f3a7fc640 1 -- 172.21.15.114:0/2826605364 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f2c003520 con 0x7f7f3402b4c0 2024-03-29T11:26:01.225 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.239+0000 7f7f3a7fc640 1 -- 172.21.15.114:0/2826605364 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3404413960 0 0) 0x7f7f2c003520 con 0x7f7f3402b4c0 2024-03-29T11:26:01.225 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.239+0000 7f7f3a7fc640 1 -- 172.21.15.114:0/2826605364 >> v1:172.21.15.114:6790/0 conn(0x7f7f34006890 legacy=0x7f7f340c0a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:26:01.225 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.239+0000 7f7f3a7fc640 1 -- 172.21.15.114:0/2826605364 >> v1:172.21.15.176:6789/0 conn(0x7f7f34006c70 legacy=0x7f7f340c6d40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:26:01.228 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.242+0000 7f7f3a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:26:01.228 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.242+0000 7f7f3a7fc640 1 monclient: found mon.noname-a 2024-03-29T11:26:01.228 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.242+0000 7f7f3a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:01.228 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.242+0000 7f7f3a7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.114:6789/0 2024-03-29T11:26:01.228 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.242+0000 7f7f3a7fc640 1 -- 172.21.15.114:0/2826605364 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f3402ad30 con 0x7f7f3402b4c0 2024-03-29T11:26:01.228 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.242+0000 7f7f3a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:26:01.229 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.242+0000 7f7f3a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:26:01.229 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.242+0000 7f7f3a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:01.229 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.242+0000 7f7f3a7fc640 1 -- 172.21.15.114:0/2826605364 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f7f3c04f730 con 0x7f7f3402b4c0 2024-03-29T11:26:01.229 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.242+0000 7f7f40e51640 5 monclient: authenticate success, global_id 4407 2024-03-29T11:26:01.229 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.242+0000 7f7f40e51640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:01.229 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.242+0000 7f7f3a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:01.229 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.243+0000 7f7f3a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:26:01.229 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.243+0000 7f7f3a7fc640 10 monclient: dump: 2024-03-29T11:26:01.229 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:26:01.229 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:26:01.229 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:26:01.229 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:26:01.229 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:01.229 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:26:01.230 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:26:01.230 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:26:01.230 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:26:01.230 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:26:01.230 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.243+0000 7f7f40e51640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:01.231 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.245+0000 7f7f3a7fc640 1 -- 172.21.15.114:0/2826605364 <== mon.0 v1:172.21.15.114:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7f3c075f10 con 0x7f7f3402b4c0 2024-03-29T11:26:01.231 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.245+0000 7f7f3a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:01.231 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.245+0000 7f7f3a7fc640 1 -- 172.21.15.114:0/2826605364 <== mon.0 v1:172.21.15.114:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f7f3c0758b0 con 0x7f7f3402b4c0 2024-03-29T11:26:01.231 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.245+0000 7f7f40e51640 10 monclient: get_monmap_and_config success 2024-03-29T11:26:01.231 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.245+0000 7f7f40e51640 4 set_mon_vals no callback set 2024-03-29T11:26:01.232 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.245+0000 7f7f3a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:01.232 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.246+0000 7f7f3a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:26:01.232 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.246+0000 7f7f3a7fc640 10 monclient: dump: 2024-03-29T11:26:01.232 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:26:01.232 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:26:01.232 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:26:01.232 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:26:01.232 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:01.232 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:26:01.232 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:26:01.232 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:26:01.232 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:26:01.232 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:26:01.232 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.246+0000 7f7f40e51640 10 monclient: shutdown 2024-03-29T11:26:01.233 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.246+0000 7f7f40e51640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:01.233 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.246+0000 7f7f40e51640 1 -- 172.21.15.114:0/2826605364 >> v1:172.21.15.114:6789/0 conn(0x7f7f3402b4c0 legacy=0x7f7f340cf0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:01.234 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.248+0000 7f7f40e51640 1 -- 172.21.15.114:0/2826605364 shutdown_connections 2024-03-29T11:26:01.234 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.249+0000 7f7f40e51640 1 -- 172.21.15.114:0/2826605364 >> 172.21.15.114:0/2826605364 conn(0x7f7f340058f0 msgr2=0x7f7f34005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:01.234 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.249+0000 7f7f40e51640 1 -- 172.21.15.114:0/2826605364 shutdown_connections 2024-03-29T11:26:01.234 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.249+0000 7f7f40e51640 1 -- 172.21.15.114:0/2826605364 wait complete. 2024-03-29T11:26:01.238 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.250+0000 7f7f40e51640 10 monclient: build_initial_monmap 2024-03-29T11:26:01.238 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.250+0000 7f7f40e51640 10 monclient: monmap: 2024-03-29T11:26:01.238 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:26:01.238 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:26:01.238 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T11:26:01.238 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T11:26:01.238 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:01.238 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:26:01.238 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:26:01.238 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T11:26:01.238 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T11:26:01.239 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:26:01.239 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.251+0000 7f7f40e51640 1 librados: starting msgr at 2024-03-29T11:26:01.239 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.251+0000 7f7f40e51640 1 librados: starting objecter 2024-03-29T11:26:01.239 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.251+0000 7f7f40e51640 1 Processor -- start 2024-03-29T11:26:01.240 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.253+0000 7f7f40e51640 1 -- start start 2024-03-29T11:26:01.240 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.253+0000 7f7f40e51640 1 librados: setting wanted keys 2024-03-29T11:26:01.240 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.253+0000 7f7f40e51640 1 librados: calling monclient init 2024-03-29T11:26:01.240 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.253+0000 7f7f40e51640 10 monclient: init 2024-03-29T11:26:01.240 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.254+0000 7f7f40e51640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:01.240 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.254+0000 7f7f40e51640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:26:01.241 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.254+0000 7f7f40e51640 10 monclient(hunting): picked mon.noname-a con 0x7f7f34006890 addr v1:172.21.15.114:6789/0 2024-03-29T11:26:01.241 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.254+0000 7f7f40e51640 10 monclient(hunting): picked mon.noname-c con 0x7f7f3402b4c0 addr v1:172.21.15.114:6790/0 2024-03-29T11:26:01.241 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.254+0000 7f7f40e51640 10 monclient(hunting): picked mon.noname-b con 0x7f7f340c2710 addr v1:172.21.15.176:6789/0 2024-03-29T11:26:01.241 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.254+0000 7f7f40e51640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f340b0f50 con 0x7f7f34006890 2024-03-29T11:26:01.241 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.254+0000 7f7f40e51640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f340b1230 con 0x7f7f340c2710 2024-03-29T11:26:01.241 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.254+0000 7f7f40e51640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f340b1510 con 0x7f7f3402b4c0 2024-03-29T11:26:01.241 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.254+0000 7f7f40e51640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:01.241 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.254+0000 7f7f40e51640 10 monclient(hunting): authenticate will time out at 1231.436749s 2024-03-29T11:26:01.244 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.258+0000 7f7f3b7fe640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f7f34006890 0x7f7f340afcf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:37797/0 (socket says 172.21.15.114:37797) 2024-03-29T11:26:01.244 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.259+0000 7f7f3b7fe640 1 -- 172.21.15.114:0/964970367 learned_addr learned my addr 172.21.15.114:0/964970367 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:26:01.246 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.261+0000 7f7f27fff640 10 client.?.objecter ms_handle_connect 0x7f7f340c2710 2024-03-29T11:26:01.246 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.261+0000 7f7f27fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:01.246 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.261+0000 7f7f27fff640 10 client.?.objecter ms_handle_connect 0x7f7f34006890 2024-03-29T11:26:01.246 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.261+0000 7f7f27fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:01.246 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.261+0000 7f7f27fff640 1 -- 172.21.15.114:0/964970367 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1421589272 0 0) 0x7f7f340b1230 con 0x7f7f340c2710 2024-03-29T11:26:01.246 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.261+0000 7f7f27fff640 10 monclient(hunting): _negotiate 2024-03-29T11:26:01.246 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.261+0000 7f7f27fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:01.247 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.261+0000 7f7f27fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:01.247 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.261+0000 7f7f27fff640 10 monclient(hunting): my global_id is 4402 2024-03-29T11:26:01.247 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.261+0000 7f7f27fff640 1 -- 172.21.15.114:0/964970367 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f20004eb0 con 0x7f7f340c2710 2024-03-29T11:26:01.247 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.261+0000 7f7f27fff640 1 -- 172.21.15.114:0/964970367 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2079488485 0 0) 0x7f7f340b0f50 con 0x7f7f34006890 2024-03-29T11:26:01.247 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.261+0000 7f7f27fff640 10 monclient(hunting): _negotiate 2024-03-29T11:26:01.247 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.261+0000 7f7f27fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:01.247 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.261+0000 7f7f27fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:01.247 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.261+0000 7f7f27fff640 10 monclient(hunting): my global_id is 4410 2024-03-29T11:26:01.247 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.261+0000 7f7f27fff640 1 -- 172.21.15.114:0/964970367 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f340b1230 con 0x7f7f34006890 2024-03-29T11:26:01.248 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.261+0000 7f7f27fff640 1 -- 172.21.15.114:0/964970367 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4190981987 0 0) 0x7f7f20004eb0 con 0x7f7f340c2710 2024-03-29T11:26:01.248 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.261+0000 7f7f27fff640 1 -- 172.21.15.114:0/964970367 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7f340b0f50 con 0x7f7f340c2710 2024-03-29T11:26:01.248 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.261+0000 7f7f27fff640 1 -- 172.21.15.114:0/964970367 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f7f3c04f400 con 0x7f7f340c2710 2024-03-29T11:26:01.248 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.261+0000 7f7f27fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:01.248 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.261+0000 7f7f27fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:26:01.248 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.261+0000 7f7f27fff640 10 monclient(hunting): dump: 2024-03-29T11:26:01.248 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:26:01.248 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:26:01.248 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:26:01.248 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:26:01.248 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:01.248 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:26:01.249 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:26:01.249 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:26:01.249 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:26:01.249 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:26:01.249 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.261+0000 7f7f27fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:26:01.249 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.261+0000 7f7f27fff640 1 -- 172.21.15.114:0/964970367 <== mon.1 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 79753573 0 0) 0x7f7f340b0f50 con 0x7f7f340c2710 2024-03-29T11:26:01.249 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.261+0000 7f7f27fff640 1 -- 172.21.15.114:0/964970367 >> v1:172.21.15.114:6790/0 conn(0x7f7f3402b4c0 legacy=0x7f7f340b21a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:01.249 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.261+0000 7f7f27fff640 1 -- 172.21.15.114:0/964970367 >> v1:172.21.15.114:6789/0 conn(0x7f7f34006890 legacy=0x7f7f340afcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:01.249 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.261+0000 7f7f27fff640 10 monclient: _finish_hunting 0 2024-03-29T11:26:01.249 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.261+0000 7f7f27fff640 1 monclient: found mon.b 2024-03-29T11:26:01.249 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.262+0000 7f7f27fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:01.249 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.262+0000 7f7f27fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:26:01.249 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.262+0000 7f7f27fff640 1 -- 172.21.15.114:0/964970367 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f340b17f0 con 0x7f7f340c2710 2024-03-29T11:26:01.249 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.262+0000 7f7f27fff640 10 monclient: _finish_auth 0 2024-03-29T11:26:01.249 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.262+0000 7f7f27fff640 10 monclient: _check_auth_tickets 2024-03-29T11:26:01.250 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.262+0000 7f7f27fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:01.250 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.262+0000 7f7f40e51640 5 monclient: authenticate success, global_id 4402 2024-03-29T11:26:01.250 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.262+0000 7f7f40e51640 10 monclient: _renew_subs 2024-03-29T11:26:01.250 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.262+0000 7f7f40e51640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:26:01.250 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.262+0000 7f7f40e51640 1 -- 172.21.15.114:0/964970367 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f340b1c10 con 0x7f7f340c2710 2024-03-29T11:26:01.250 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.262+0000 7f7f40e51640 10 client.4402.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:26:01.250 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.262+0000 7f7f40e51640 10 monclient: _renew_subs 2024-03-29T11:26:01.250 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.262+0000 7f7f40e51640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:26:01.250 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.262+0000 7f7f40e51640 1 -- 172.21.15.114:0/964970367 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f3416ceb0 con 0x7f7f340c2710 2024-03-29T11:26:01.250 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.262+0000 7f7f40e51640 1 librados: init done 2024-03-29T11:26:01.250 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.262+0000 7f7f27fff640 1 -- 172.21.15.114:0/964970367 <== mon.1 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7f3c04fdd0 con 0x7f7f340c2710 2024-03-29T11:26:01.251 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.262+0000 7f7f27fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:01.251 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.262+0000 7f7f27fff640 1 -- 172.21.15.114:0/964970367 <== mon.1 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f7f3c067bf0 con 0x7f7f340c2710 2024-03-29T11:26:01.252 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.262+0000 7f7f27fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:01.252 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.262+0000 7f7f27fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:26:01.252 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.262+0000 7f7f27fff640 10 monclient: dump: 2024-03-29T11:26:01.252 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:26:01.252 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:26:01.252 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:26:01.252 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:26:01.252 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:01.252 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:26:01.252 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:26:01.252 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:26:01.252 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:26:01.252 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:26:01.253 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.263+0000 7f7f40e51640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:01.253 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.265+0000 7f7f397fa640 4 set_mon_vals no callback set 2024-03-29T11:26:01.253 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.266+0000 7f7f40e51640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:01.253 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.267+0000 7f7f27fff640 1 -- 172.21.15.114:0/964970367 <== mon.1 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1594292461 0 0) 0x7f7f3c067d50 con 0x7f7f340c2710 2024-03-29T11:26:01.254 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.267+0000 7f7f40e51640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:26:01.254 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.267+0000 7f7f40e51640 1 -- 172.21.15.114:0/964970367 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f340b1e00 con 0x7f7f340c2710 2024-03-29T11:26:01.254 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.268+0000 7f7f27fff640 1 -- 172.21.15.114:0/964970367 <== mon.1 v1:172.21.15.176:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3238987313 0 0) 0x7f7f3c0ff560 con 0x7f7f340c2710 2024-03-29T11:26:01.254 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.268+0000 7f7f27fff640 10 client.4402.objecter ms_dispatch 0x7f7f3c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:26:01.254 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.268+0000 7f7f27fff640 3 client.4402.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:26:01.254 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.268+0000 7f7f27fff640 3 client.4402.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:26:01.254 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.268+0000 7f7f27fff640 20 client.4402.objecter dump_active .. 0 homeless 2024-03-29T11:26:01.254 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.268+0000 7f7f3b7fe640 10 monclient: get_auth_request con 0x7f7f2006da60 auth_method 0 2024-03-29T11:26:01.255 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.269+0000 7f7f27fff640 10 client.4402.objecter ms_handle_connect 0x7f7f2006da60 2024-03-29T11:26:01.261 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.275+0000 7f7f27fff640 1 -- 172.21.15.114:0/964970367 <== mon.1 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7f3c0cae40 con 0x7f7f340c2710 2024-03-29T11:26:01.261 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.275+0000 7f7f27fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:01.261 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.275+0000 7f7f27fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:26:01.589 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.603+0000 7f7f40e51640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:01.589 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.603+0000 7f7f40e51640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:01.589 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.603+0000 7f7f40e51640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:26:01.589 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.603+0000 7f7f40e51640 1 -- 172.21.15.114:0/964970367 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7f3402a520 con 0x7f7f340c2710 2024-03-29T11:26:01.592 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.606+0000 7f7f27fff640 1 -- 172.21.15.114:0/964970367 <== mon.1 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2858224613) 0x7f7f3c0ccd70 con 0x7f7f340c2710 2024-03-29T11:26:01.592 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.606+0000 7f7f27fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:01.592 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.606+0000 7f7f27fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:26:01.592 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T11:26:01.592 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"b0de2b9b-c45a-44c7-b6ee-e83b39a1e108","created":"2024-03-29T11:20:49.420660+0000","modified":"2024-03-29T11:20:51.700167+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:51.700167+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a64785d3-6b11-499a-b706-cc827ab96a74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5882dadb-e7a8-4604-ba9a-418013c106c8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d7e8c409-bc66-4e4a-bbdb-420bad31388e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0682989-3595-4cd6-90a5-ab9111c278e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"67068120-e3ff-4817-bd06-44feb7479a35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09e8b-8a85-4997-bdea-75d754e6ee67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2f0d9ead-f47f-4cb3-92a4-0a5ad23cd831","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fdb6a25f-6bd8-4c65-9b9b-b460ba7673a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:26:01.596 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.610+0000 7f7f40e51640 10 librados: watch_flush enter 2024-03-29T11:26:01.596 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.610+0000 7f7f40e51640 10 librados: watch_flush exit 2024-03-29T11:26:01.596 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.610+0000 7f7f40e51640 20 client.4402.objecter shutdown clearing up homeless session... 2024-03-29T11:26:01.596 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.610+0000 7f7f40e51640 10 client.4402.objecter successfully canceled tick 2024-03-29T11:26:01.596 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.610+0000 7f7f40e51640 1 -- 172.21.15.114:0/964970367 >> v1:172.21.15.114:6816/4038755055 conn(0x7f7f2006da60 legacy=0x7f7f2006ff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:01.596 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.610+0000 7f7f40e51640 10 monclient: shutdown 2024-03-29T11:26:01.596 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.610+0000 7f7f40e51640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:01.596 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.610+0000 7f7f40e51640 1 -- 172.21.15.114:0/964970367 >> v1:172.21.15.176:6789/0 conn(0x7f7f340c2710 legacy=0x7f7f3416b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:01.597 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.612+0000 7f7f40e51640 1 -- 172.21.15.114:0/964970367 shutdown_connections 2024-03-29T11:26:01.597 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.612+0000 7f7f40e51640 1 -- 172.21.15.114:0/964970367 >> 172.21.15.114:0/964970367 conn(0x7f7f340058f0 msgr2=0x7f7f34038130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:01.597 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.612+0000 7f7f40e51640 1 -- 172.21.15.114:0/964970367 shutdown_connections 2024-03-29T11:26:01.597 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.612+0000 7f7f40e51640 1 -- 172.21.15.114:0/964970367 wait complete. 2024-03-29T11:26:01.598 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:01.612+0000 7f7f40e51640 1 librados: shutdown 2024-03-29T11:26:01.614 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:26:08.422 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:26:08.577 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.592+0000 7f44ec58c640 10 monclient: get_monmap_and_config 2024-03-29T11:26:08.578 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.592+0000 7f44ec58c640 10 monclient: build_initial_monmap 2024-03-29T11:26:08.578 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.592+0000 7f44ec58c640 10 monclient: monmap: 2024-03-29T11:26:08.578 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:26:08.578 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:26:08.578 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:26:08.593298+0000 2024-03-29T11:26:08.578 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:26:08.593298+0000 2024-03-29T11:26:08.578 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:08.578 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:26:08.578 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:26:08.578 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T11:26:08.578 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T11:26:08.578 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:26:08.579 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.594+0000 7f44ec58c640 1 Processor -- start 2024-03-29T11:26:08.580 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.594+0000 7f44ec58c640 1 -- start start 2024-03-29T11:26:08.580 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.594+0000 7f44ec58c640 10 monclient: init 2024-03-29T11:26:08.584 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.596+0000 7f44ec58c640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:08.584 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.596+0000 7f44ec58c640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:26:08.584 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.596+0000 7f44ec58c640 10 monclient(hunting): picked mon.noname-b con 0x7f44e41457f0 addr v1:172.21.15.114:6790/0 2024-03-29T11:26:08.584 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.596+0000 7f44ec58c640 10 monclient(hunting): picked mon.noname-a con 0x7f44e413f650 addr v1:172.21.15.114:6789/0 2024-03-29T11:26:08.584 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.596+0000 7f44ec58c640 10 monclient(hunting): picked mon.noname-c con 0x7f44e40a9d70 addr v1:172.21.15.176:6789/0 2024-03-29T11:26:08.584 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.596+0000 7f44ec58c640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44e40aaef0 con 0x7f44e413f650 2024-03-29T11:26:08.584 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.596+0000 7f44ec58c640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44e4145bd0 con 0x7f44e40a9d70 2024-03-29T11:26:08.584 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.596+0000 7f44ec58c640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44e4145eb0 con 0x7f44e41457f0 2024-03-29T11:26:08.585 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.596+0000 7f44ec58c640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:08.585 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.596+0000 7f44ec58c640 10 monclient(hunting): authenticate will time out at 1238.778427s 2024-03-29T11:26:08.587 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.601+0000 7f44e9b27640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f44e413f650 0x7f44e40b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:33839/0 (socket says 172.21.15.114:33839) 2024-03-29T11:26:08.587 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.601+0000 7f44e9b27640 1 -- 172.21.15.114:0/594208204 learned_addr learned my addr 172.21.15.114:0/594208204 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:26:08.593 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.604+0000 7f44e9326640 1 -- 172.21.15.114:0/594208204 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2764491784 0 0) 0x7f44e40aaef0 con 0x7f44e413f650 2024-03-29T11:26:08.593 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.604+0000 7f44e9326640 10 monclient(hunting): _negotiate 2024-03-29T11:26:08.593 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.604+0000 7f44e9326640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:08.593 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.604+0000 7f44e9326640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:08.593 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.604+0000 7f44e9326640 10 monclient(hunting): my global_id is 4413 2024-03-29T11:26:08.593 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.604+0000 7f44e9326640 1 -- 172.21.15.114:0/594208204 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44d0003520 con 0x7f44e413f650 2024-03-29T11:26:08.594 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.604+0000 7f44ea328640 1 -- 172.21.15.114:0/594208204 >> v1:172.21.15.114:6790/0 conn(0x7f44e41457f0 legacy=0x7f44e413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2024-03-29T11:26:08.594 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.604+0000 7f44ea328640 1 -- 172.21.15.114:0/594208204 >> v1:172.21.15.114:6790/0 conn(0x7f44e41457f0 legacy=0x7f44e413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T11:26:08.594 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.604+0000 7f44ea328640 1 --1- 172.21.15.114:0/594208204 >> v1:172.21.15.114:6790/0 conn(0x7f44e41457f0 0x7f44e413ef40 :-1 s=OPENED pgs=517951 cs=1 l=1).handle_message read tag failed 2024-03-29T11:26:08.594 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.604+0000 7f44ea328640 1 --1- 172.21.15.114:0/594208204 >> v1:172.21.15.114:6790/0 conn(0x7f44e41457f0 0x7f44e413ef40 :-1 s=OPENED pgs=517951 cs=1 l=1).fault on lossy channel, failing 2024-03-29T11:26:08.594 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.604+0000 7f44e9326640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.114:6790/0 2024-03-29T11:26:08.594 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.605+0000 7f44e9326640 1 -- 172.21.15.114:0/594208204 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2139925474 0 0) 0x7f44d0003520 con 0x7f44e413f650 2024-03-29T11:26:08.594 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.605+0000 7f44e9326640 1 -- 172.21.15.114:0/594208204 >> v1:172.21.15.114:6790/0 conn(0x7f44e41457f0 legacy=0x7f44e413ef40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T11:26:08.594 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.605+0000 7f44e9326640 1 -- 172.21.15.114:0/594208204 >> v1:172.21.15.176:6789/0 conn(0x7f44e40a9d70 legacy=0x7f44e414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:08.594 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.605+0000 7f44e9326640 10 monclient: _finish_hunting 0 2024-03-29T11:26:08.594 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.605+0000 7f44e9326640 1 monclient: found mon.noname-a 2024-03-29T11:26:08.594 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.605+0000 7f44e9326640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:08.594 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.605+0000 7f44e9326640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.114:6789/0 2024-03-29T11:26:08.594 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.605+0000 7f44e9326640 1 -- 172.21.15.114:0/594208204 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44e4146340 con 0x7f44e413f650 2024-03-29T11:26:08.594 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.605+0000 7f44e9326640 10 monclient: _finish_auth 0 2024-03-29T11:26:08.595 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.605+0000 7f44e9326640 10 monclient: _check_auth_tickets 2024-03-29T11:26:08.595 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.605+0000 7f44e9326640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:08.595 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.605+0000 7f44ec58c640 5 monclient: authenticate success, global_id 4413 2024-03-29T11:26:08.595 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.605+0000 7f44ec58c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:08.595 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.605+0000 7f44e9326640 1 -- 172.21.15.114:0/594208204 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f44e0001670 con 0x7f44e413f650 2024-03-29T11:26:08.595 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.605+0000 7f44e9326640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:08.595 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.605+0000 7f44e9326640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:26:08.595 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.605+0000 7f44e9326640 10 monclient: dump: 2024-03-29T11:26:08.595 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:26:08.595 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:26:08.595 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:26:08.595 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:26:08.595 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:08.595 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:26:08.595 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:26:08.596 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:26:08.596 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:26:08.596 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:26:08.596 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.605+0000 7f44ec58c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:08.596 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.605+0000 7f44e9326640 1 -- 172.21.15.114:0/594208204 <== mon.0 v1:172.21.15.114:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f44e0001850 con 0x7f44e413f650 2024-03-29T11:26:08.596 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.605+0000 7f44e9326640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:08.596 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.605+0000 7f44ec58c640 10 monclient: get_monmap_and_config success 2024-03-29T11:26:08.596 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.605+0000 7f44ec58c640 4 set_mon_vals no callback set 2024-03-29T11:26:08.596 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.605+0000 7f44e9326640 1 -- 172.21.15.114:0/594208204 <== mon.0 v1:172.21.15.114:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f44e0001670 con 0x7f44e413f650 2024-03-29T11:26:08.596 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.605+0000 7f44e9326640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:08.596 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.605+0000 7f44e9326640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:26:08.596 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.605+0000 7f44e9326640 10 monclient: dump: 2024-03-29T11:26:08.596 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:26:08.596 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:26:08.596 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:26:08.597 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:26:08.597 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:08.597 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:26:08.597 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:26:08.597 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:26:08.597 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:26:08.597 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:26:08.597 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.606+0000 7f44ec58c640 10 monclient: shutdown 2024-03-29T11:26:08.597 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.606+0000 7f44ec58c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:08.597 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.606+0000 7f44ec58c640 1 -- 172.21.15.114:0/594208204 >> v1:172.21.15.114:6789/0 conn(0x7f44e413f650 legacy=0x7f44e40b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:08.597 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.606+0000 7f44ec58c640 1 -- 172.21.15.114:0/594208204 shutdown_connections 2024-03-29T11:26:08.598 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.606+0000 7f44ec58c640 1 -- 172.21.15.114:0/594208204 >> 172.21.15.114:0/594208204 conn(0x7f44e4066a00 msgr2=0x7f44e4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:08.599 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.609+0000 7f44ec58c640 1 -- 172.21.15.114:0/594208204 shutdown_connections 2024-03-29T11:26:08.599 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.609+0000 7f44ec58c640 1 -- 172.21.15.114:0/594208204 wait complete. 2024-03-29T11:26:08.599 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.610+0000 7f44ec58c640 10 monclient: build_initial_monmap 2024-03-29T11:26:08.599 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.610+0000 7f44ec58c640 10 monclient: monmap: 2024-03-29T11:26:08.599 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:26:08.599 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:26:08.599 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T11:26:08.600 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T11:26:08.600 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:08.600 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:26:08.600 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:26:08.600 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T11:26:08.600 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T11:26:08.600 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:26:08.600 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.611+0000 7f44ec58c640 1 librados: starting msgr at 2024-03-29T11:26:08.600 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.611+0000 7f44ec58c640 1 librados: starting objecter 2024-03-29T11:26:08.600 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.611+0000 7f44ec58c640 1 Processor -- start 2024-03-29T11:26:08.600 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.611+0000 7f44ec58c640 1 -- start start 2024-03-29T11:26:08.600 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.611+0000 7f44ec58c640 1 librados: setting wanted keys 2024-03-29T11:26:08.600 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.611+0000 7f44ec58c640 1 librados: calling monclient init 2024-03-29T11:26:08.600 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.611+0000 7f44ec58c640 10 monclient: init 2024-03-29T11:26:08.601 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.612+0000 7f44ec58c640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:08.601 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.612+0000 7f44ec58c640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:26:08.601 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.615+0000 7f44ec58c640 10 monclient(hunting): picked mon.noname-a con 0x7f44e40a9d70 addr v1:172.21.15.114:6789/0 2024-03-29T11:26:08.603 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.615+0000 7f44ec58c640 10 monclient(hunting): picked mon.noname-b con 0x7f44e413f650 addr v1:172.21.15.176:6789/0 2024-03-29T11:26:08.606 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.618+0000 7f44ec58c640 10 monclient(hunting): picked mon.noname-c con 0x7f44e41457f0 addr v1:172.21.15.114:6790/0 2024-03-29T11:26:08.606 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.618+0000 7f44ec58c640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44e40acd00 con 0x7f44e40a9d70 2024-03-29T11:26:08.606 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.618+0000 7f44ec58c640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44e40acf70 con 0x7f44e413f650 2024-03-29T11:26:08.606 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.618+0000 7f44ec58c640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44e40ad250 con 0x7f44e41457f0 2024-03-29T11:26:08.606 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.618+0000 7f44ec58c640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:08.607 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.618+0000 7f44ec58c640 10 monclient(hunting): authenticate will time out at 1238.800253s 2024-03-29T11:26:08.607 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.618+0000 7f44ea328640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f44e40a9d70 0x7f44e41d7820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:38013/0 (socket says 172.21.15.114:38013) 2024-03-29T11:26:08.607 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.618+0000 7f44ea328640 1 -- 172.21.15.114:0/4033615990 learned_addr learned my addr 172.21.15.114:0/4033615990 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:26:08.607 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.618+0000 7f44daffd640 10 client.?.objecter ms_handle_connect 0x7f44e40a9d70 2024-03-29T11:26:08.607 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.619+0000 7f44daffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:08.607 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.619+0000 7f44daffd640 10 client.?.objecter ms_handle_connect 0x7f44e41457f0 2024-03-29T11:26:08.607 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.619+0000 7f44daffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:08.607 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.619+0000 7f44daffd640 10 client.?.objecter ms_handle_connect 0x7f44e413f650 2024-03-29T11:26:08.607 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.619+0000 7f44daffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:08.607 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.619+0000 7f44daffd640 1 -- 172.21.15.114:0/4033615990 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3982743483 0 0) 0x7f44e40acd00 con 0x7f44e40a9d70 2024-03-29T11:26:08.607 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.619+0000 7f44daffd640 10 monclient(hunting): _negotiate 2024-03-29T11:26:08.607 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.619+0000 7f44daffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:08.607 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.619+0000 7f44daffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:08.607 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.619+0000 7f44daffd640 10 monclient(hunting): my global_id is 4416 2024-03-29T11:26:08.607 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.619+0000 7f44daffd640 1 -- 172.21.15.114:0/4033615990 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44c8004eb0 con 0x7f44e40a9d70 2024-03-29T11:26:08.608 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.619+0000 7f44daffd640 1 -- 172.21.15.114:0/4033615990 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1323478843 0 0) 0x7f44e40acf70 con 0x7f44e413f650 2024-03-29T11:26:08.608 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.619+0000 7f44daffd640 10 monclient(hunting): _negotiate 2024-03-29T11:26:08.608 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.619+0000 7f44daffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:08.608 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.619+0000 7f44daffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:08.608 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.619+0000 7f44daffd640 10 monclient(hunting): my global_id is 4405 2024-03-29T11:26:08.608 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.619+0000 7f44daffd640 1 -- 172.21.15.114:0/4033615990 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44e40acd00 con 0x7f44e413f650 2024-03-29T11:26:08.608 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.620+0000 7f44daffd640 1 -- 172.21.15.114:0/4033615990 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3889887705 0 0) 0x7f44e40ad250 con 0x7f44e41457f0 2024-03-29T11:26:08.608 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.620+0000 7f44daffd640 10 monclient(hunting): _negotiate 2024-03-29T11:26:08.608 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.620+0000 7f44daffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:08.608 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.620+0000 7f44daffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:08.608 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.620+0000 7f44daffd640 10 monclient(hunting): my global_id is 4430 2024-03-29T11:26:08.608 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.620+0000 7f44daffd640 1 -- 172.21.15.114:0/4033615990 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44e40acf70 con 0x7f44e41457f0 2024-03-29T11:26:08.608 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.620+0000 7f44daffd640 1 -- 172.21.15.114:0/4033615990 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1197269897 0 0) 0x7f44c8004eb0 con 0x7f44e40a9d70 2024-03-29T11:26:08.608 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.620+0000 7f44daffd640 1 -- 172.21.15.114:0/4033615990 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f44e40ad250 con 0x7f44e40a9d70 2024-03-29T11:26:08.609 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.620+0000 7f44daffd640 1 -- 172.21.15.114:0/4033615990 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 53654259 0 0) 0x7f44e40acd00 con 0x7f44e413f650 2024-03-29T11:26:08.609 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.620+0000 7f44daffd640 1 -- 172.21.15.114:0/4033615990 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f44c8004eb0 con 0x7f44e413f650 2024-03-29T11:26:08.609 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.620+0000 7f44daffd640 1 -- 172.21.15.114:0/4033615990 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f44d40018c0 con 0x7f44e40a9d70 2024-03-29T11:26:08.609 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.620+0000 7f44daffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:08.609 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.620+0000 7f44daffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:26:08.609 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.620+0000 7f44daffd640 10 monclient(hunting): dump: 2024-03-29T11:26:08.609 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:26:08.609 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:26:08.609 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:26:08.609 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:26:08.609 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:08.609 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:26:08.609 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:26:08.609 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:26:08.609 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:26:08.610 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:26:08.610 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.620+0000 7f44daffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:26:08.610 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.620+0000 7f44daffd640 1 -- 172.21.15.114:0/4033615990 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f44e0001c20 con 0x7f44e413f650 2024-03-29T11:26:08.610 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.620+0000 7f44daffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:08.610 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.620+0000 7f44daffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:26:08.610 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.620+0000 7f44daffd640 10 monclient(hunting): dump: 2024-03-29T11:26:08.610 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:26:08.610 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:26:08.610 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:26:08.610 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:26:08.610 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:08.610 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:26:08.610 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:26:08.610 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:26:08.610 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:26:08.611 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:26:08.613 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.623+0000 7f44daffd640 1 -- 172.21.15.114:0/4033615990 <== mon.1 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 93388404 0 0) 0x7f44c8004eb0 con 0x7f44e413f650 2024-03-29T11:26:08.613 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.623+0000 7f44daffd640 1 -- 172.21.15.114:0/4033615990 >> v1:172.21.15.114:6790/0 conn(0x7f44e41457f0 legacy=0x7f44e41e2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:08.614 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.623+0000 7f44daffd640 1 -- 172.21.15.114:0/4033615990 >> v1:172.21.15.114:6789/0 conn(0x7f44e40a9d70 legacy=0x7f44e41d7820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:08.614 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.623+0000 7f44daffd640 10 monclient: _finish_hunting 0 2024-03-29T11:26:08.614 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.623+0000 7f44daffd640 1 monclient: found mon.b 2024-03-29T11:26:08.614 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.623+0000 7f44daffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:08.614 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.623+0000 7f44daffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:26:08.614 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.623+0000 7f44daffd640 1 -- 172.21.15.114:0/4033615990 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44e4138d30 con 0x7f44e413f650 2024-03-29T11:26:08.614 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.623+0000 7f44daffd640 10 monclient: _finish_auth 0 2024-03-29T11:26:08.615 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.623+0000 7f44daffd640 10 monclient: _check_auth_tickets 2024-03-29T11:26:08.615 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.623+0000 7f44daffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:08.615 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.623+0000 7f44ec58c640 5 monclient: authenticate success, global_id 4405 2024-03-29T11:26:08.615 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.623+0000 7f44ec58c640 10 monclient: _renew_subs 2024-03-29T11:26:08.615 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.623+0000 7f44ec58c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:26:08.615 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.623+0000 7f44ec58c640 1 -- 172.21.15.114:0/4033615990 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44e41d7f30 con 0x7f44e413f650 2024-03-29T11:26:08.615 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.623+0000 7f44ec58c640 10 client.4405.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:26:08.615 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.623+0000 7f44ec58c640 10 monclient: _renew_subs 2024-03-29T11:26:08.615 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.623+0000 7f44ec58c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:26:08.615 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.623+0000 7f44ec58c640 1 -- 172.21.15.114:0/4033615990 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f44e40ac0c0 con 0x7f44e413f650 2024-03-29T11:26:08.615 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.623+0000 7f44ec58c640 1 librados: init done 2024-03-29T11:26:08.615 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.624+0000 7f44daffd640 1 -- 172.21.15.114:0/4033615990 <== mon.1 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f44e0001220 con 0x7f44e413f650 2024-03-29T11:26:08.615 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.624+0000 7f44daffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:08.615 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.624+0000 7f44e8b25640 4 set_mon_vals no callback set 2024-03-29T11:26:08.615 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.625+0000 7f44daffd640 1 -- 172.21.15.114:0/4033615990 <== mon.1 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f44e0004c30 con 0x7f44e413f650 2024-03-29T11:26:08.616 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.625+0000 7f44daffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:08.616 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.625+0000 7f44daffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:26:08.616 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.625+0000 7f44daffd640 10 monclient: dump: 2024-03-29T11:26:08.616 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:26:08.616 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:26:08.616 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:26:08.616 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:26:08.616 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:08.616 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:26:08.616 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:26:08.616 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:26:08.616 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:26:08.616 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:26:08.617 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.626+0000 7f44daffd640 1 -- 172.21.15.114:0/4033615990 <== mon.1 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1594292461 0 0) 0x7f44e001ac00 con 0x7f44e413f650 2024-03-29T11:26:08.617 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.626+0000 7f44daffd640 1 -- 172.21.15.114:0/4033615990 <== mon.1 v1:172.21.15.176:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3238987313 0 0) 0x7f44e0084a90 con 0x7f44e413f650 2024-03-29T11:26:08.617 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.626+0000 7f44daffd640 10 client.4405.objecter ms_dispatch 0x7f44e4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:26:08.617 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.626+0000 7f44daffd640 3 client.4405.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:26:08.617 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.626+0000 7f44daffd640 3 client.4405.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:26:08.617 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.626+0000 7f44daffd640 20 client.4405.objecter dump_active .. 0 homeless 2024-03-29T11:26:08.617 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.627+0000 7f44ea328640 10 monclient: get_auth_request con 0x7f44c806dd40 auth_method 0 2024-03-29T11:26:08.617 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.627+0000 7f44ec58c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:08.617 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.627+0000 7f44ec58c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:08.617 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.627+0000 7f44ec58c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:26:08.617 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.627+0000 7f44ec58c640 1 -- 172.21.15.114:0/4033615990 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44cc005080 con 0x7f44e413f650 2024-03-29T11:26:08.617 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.627+0000 7f44daffd640 10 client.4405.objecter ms_handle_connect 0x7f44c806dd40 2024-03-29T11:26:08.622 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.635+0000 7f44daffd640 1 -- 172.21.15.114:0/4033615990 <== mon.1 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f44e0050370 con 0x7f44e413f650 2024-03-29T11:26:08.622 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.635+0000 7f44daffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:08.622 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.635+0000 7f44daffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:26:08.967 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.982+0000 7f44ec58c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:08.967 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.982+0000 7f44ec58c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:08.968 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.982+0000 7f44ec58c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:26:08.968 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.982+0000 7f44ec58c640 1 -- 172.21.15.114:0/4033615990 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f44cc005330 con 0x7f44e413f650 2024-03-29T11:26:08.968 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.983+0000 7f44daffd640 1 -- 172.21.15.114:0/4033615990 <== mon.1 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2858224613) 0x7f44e00522a0 con 0x7f44e413f650 2024-03-29T11:26:08.968 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.983+0000 7f44daffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:08.968 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.983+0000 7f44daffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:26:08.969 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T11:26:08.969 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"b0de2b9b-c45a-44c7-b6ee-e83b39a1e108","created":"2024-03-29T11:20:49.420660+0000","modified":"2024-03-29T11:20:51.700167+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:51.700167+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a64785d3-6b11-499a-b706-cc827ab96a74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5882dadb-e7a8-4604-ba9a-418013c106c8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d7e8c409-bc66-4e4a-bbdb-420bad31388e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0682989-3595-4cd6-90a5-ab9111c278e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"67068120-e3ff-4817-bd06-44feb7479a35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09e8b-8a85-4997-bdea-75d754e6ee67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2f0d9ead-f47f-4cb3-92a4-0a5ad23cd831","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fdb6a25f-6bd8-4c65-9b9b-b460ba7673a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:26:08.973 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.987+0000 7f44ec58c640 10 librados: watch_flush enter 2024-03-29T11:26:08.973 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.987+0000 7f44ec58c640 10 librados: watch_flush exit 2024-03-29T11:26:08.973 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.987+0000 7f44ec58c640 20 client.4405.objecter shutdown clearing up homeless session... 2024-03-29T11:26:08.973 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.987+0000 7f44ec58c640 10 client.4405.objecter successfully canceled tick 2024-03-29T11:26:08.973 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.988+0000 7f44ec58c640 1 -- 172.21.15.114:0/4033615990 >> v1:172.21.15.114:6816/4038755055 conn(0x7f44c806dd40 legacy=0x7f44c8070200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:08.973 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.988+0000 7f44ec58c640 10 monclient: shutdown 2024-03-29T11:26:08.973 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.988+0000 7f44ec58c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:08.973 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.988+0000 7f44ec58c640 1 -- 172.21.15.114:0/4033615990 >> v1:172.21.15.176:6789/0 conn(0x7f44e413f650 legacy=0x7f44e40b0dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:08.974 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.988+0000 7f44ec58c640 1 -- 172.21.15.114:0/4033615990 shutdown_connections 2024-03-29T11:26:08.974 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.988+0000 7f44ec58c640 1 -- 172.21.15.114:0/4033615990 >> 172.21.15.114:0/4033615990 conn(0x7f44e4066a00 msgr2=0x7f44e4066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:08.974 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.988+0000 7f44ec58c640 1 -- 172.21.15.114:0/4033615990 shutdown_connections 2024-03-29T11:26:08.974 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.988+0000 7f44ec58c640 1 -- 172.21.15.114:0/4033615990 wait complete. 2024-03-29T11:26:08.974 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:08.988+0000 7f44ec58c640 1 librados: shutdown 2024-03-29T11:26:08.987 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:26:15.795 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:26:15.948 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.962+0000 7fbb0dbe2640 10 monclient: get_monmap_and_config 2024-03-29T11:26:15.949 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.962+0000 7fbb0dbe2640 10 monclient: build_initial_monmap 2024-03-29T11:26:15.949 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.962+0000 7fbb0dbe2640 10 monclient: monmap: 2024-03-29T11:26:15.949 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:26:15.949 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:26:15.949 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:26:15.963374+0000 2024-03-29T11:26:15.949 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:26:15.963374+0000 2024-03-29T11:26:15.949 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:15.949 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:26:15.949 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:26:15.949 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T11:26:15.949 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T11:26:15.950 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:26:15.952 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.964+0000 7fbb0dbe2640 1 Processor -- start 2024-03-29T11:26:15.952 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.964+0000 7fbb0dbe2640 1 -- start start 2024-03-29T11:26:15.952 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.964+0000 7fbb0dbe2640 10 monclient: init 2024-03-29T11:26:15.952 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.965+0000 7fbb0dbe2640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:15.952 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.965+0000 7fbb0dbe2640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:26:15.952 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.965+0000 7fbb0dbe2640 10 monclient(hunting): picked mon.noname-b con 0x7fbb080a9930 addr v1:172.21.15.114:6790/0 2024-03-29T11:26:15.952 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.965+0000 7fbb0dbe2640 10 monclient(hunting): picked mon.noname-c con 0x7fbb08149dc0 addr v1:172.21.15.176:6789/0 2024-03-29T11:26:15.952 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.965+0000 7fbb0dbe2640 10 monclient(hunting): picked mon.noname-a con 0x7fbb0814ff40 addr v1:172.21.15.114:6789/0 2024-03-29T11:26:15.953 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.965+0000 7fbb0dbe2640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb080aae60 con 0x7fbb0814ff40 2024-03-29T11:26:15.953 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.965+0000 7fbb0dbe2640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb080b4310 con 0x7fbb08149dc0 2024-03-29T11:26:15.953 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.965+0000 7fbb0dbe2640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb080afdd0 con 0x7fbb080a9930 2024-03-29T11:26:15.953 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.965+0000 7fbb0dbe2640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:15.953 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.965+0000 7fbb0dbe2640 10 monclient(hunting): authenticate will time out at 1246.147221s 2024-03-29T11:26:15.956 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.970+0000 7fbb077fe640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7fbb080a9930 0x7fbb08143a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:49981/0 (socket says 172.21.15.114:49981) 2024-03-29T11:26:15.957 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.970+0000 7fbb077fe640 1 -- 172.21.15.114:0/4074384567 learned_addr learned my addr 172.21.15.114:0/4074384567 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:26:15.960 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.973+0000 7fbb067fc640 1 -- 172.21.15.114:0/4074384567 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 901459956 0 0) 0x7fbb080aae60 con 0x7fbb0814ff40 2024-03-29T11:26:15.960 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.973+0000 7fbb067fc640 10 monclient(hunting): _negotiate 2024-03-29T11:26:15.960 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.973+0000 7fbb067fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:15.960 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.973+0000 7fbb067fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:15.960 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.973+0000 7fbb067fc640 10 monclient(hunting): my global_id is 4419 2024-03-29T11:26:15.965 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.973+0000 7fbb067fc640 1 -- 172.21.15.114:0/4074384567 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbaf0003520 con 0x7fbb0814ff40 2024-03-29T11:26:15.965 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.973+0000 7fbb067fc640 1 -- 172.21.15.114:0/4074384567 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4157607975 0 0) 0x7fbaf0003520 con 0x7fbb0814ff40 2024-03-29T11:26:15.966 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.973+0000 7fbb067fc640 1 -- 172.21.15.114:0/4074384567 >> v1:172.21.15.114:6790/0 conn(0x7fbb080a9930 legacy=0x7fbb08143a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:15.966 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.973+0000 7fbb067fc640 1 -- 172.21.15.114:0/4074384567 >> v1:172.21.15.176:6789/0 conn(0x7fbb08149dc0 legacy=0x7fbb0814a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:26:15.966 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.973+0000 7fbb067fc640 10 monclient: _finish_hunting 0 2024-03-29T11:26:15.966 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.973+0000 7fbb067fc640 1 monclient: found mon.noname-a 2024-03-29T11:26:15.966 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.973+0000 7fbb067fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:15.966 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.973+0000 7fbb067fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.114:6789/0 2024-03-29T11:26:15.966 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.973+0000 7fbb067fc640 1 -- 172.21.15.114:0/4074384567 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb0814a950 con 0x7fbb0814ff40 2024-03-29T11:26:15.966 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.973+0000 7fbb067fc640 10 monclient: _finish_auth 0 2024-03-29T11:26:15.966 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.973+0000 7fbb067fc640 10 monclient: _check_auth_tickets 2024-03-29T11:26:15.966 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.973+0000 7fbb067fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:15.966 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.973+0000 7fbb067fc640 1 -- 172.21.15.114:0/4074384567 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fbaf8001690 con 0x7fbb0814ff40 2024-03-29T11:26:15.967 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.973+0000 7fbb067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:15.967 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.973+0000 7fbb067fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:26:15.967 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.973+0000 7fbb067fc640 10 monclient: dump: 2024-03-29T11:26:15.967 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:26:15.967 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:26:15.967 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:26:15.967 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:26:15.967 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:15.967 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:26:15.967 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:26:15.967 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:26:15.967 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:26:15.967 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:26:15.967 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.974+0000 7fbb067fc640 1 -- 172.21.15.114:0/4074384567 <== mon.0 v1:172.21.15.114:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbaf8001860 con 0x7fbb0814ff40 2024-03-29T11:26:15.968 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.974+0000 7fbb067fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:15.968 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.974+0000 7fbb067fc640 1 -- 172.21.15.114:0/4074384567 <== mon.0 v1:172.21.15.114:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fbaf8001c70 con 0x7fbb0814ff40 2024-03-29T11:26:15.968 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.974+0000 7fbb067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:15.968 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.974+0000 7fbb067fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:26:15.968 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.974+0000 7fbb067fc640 10 monclient: dump: 2024-03-29T11:26:15.968 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:26:15.968 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:26:15.968 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:26:15.968 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:26:15.968 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:15.968 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:26:15.968 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:26:15.968 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:26:15.968 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:26:15.968 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:26:15.969 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.975+0000 7fbb0dbe2640 5 monclient: authenticate success, global_id 4419 2024-03-29T11:26:15.969 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.975+0000 7fbb0dbe2640 10 monclient: get_monmap_and_config success 2024-03-29T11:26:15.969 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.975+0000 7fbb0dbe2640 4 set_mon_vals no callback set 2024-03-29T11:26:15.969 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.976+0000 7fbb0dbe2640 10 monclient: shutdown 2024-03-29T11:26:15.969 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.976+0000 7fbb0dbe2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:15.969 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.976+0000 7fbb0dbe2640 1 -- 172.21.15.114:0/4074384567 >> v1:172.21.15.114:6789/0 conn(0x7fbb0814ff40 legacy=0x7fbb08152330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:15.969 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.976+0000 7fbb0dbe2640 1 -- 172.21.15.114:0/4074384567 shutdown_connections 2024-03-29T11:26:15.969 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.976+0000 7fbb0dbe2640 1 -- 172.21.15.114:0/4074384567 >> 172.21.15.114:0/4074384567 conn(0x7fbb080b35e0 msgr2=0x7fbb080b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:15.969 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.976+0000 7fbb0dbe2640 1 -- 172.21.15.114:0/4074384567 shutdown_connections 2024-03-29T11:26:15.969 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.976+0000 7fbb0dbe2640 1 -- 172.21.15.114:0/4074384567 wait complete. 2024-03-29T11:26:15.969 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.977+0000 7fbb0dbe2640 10 monclient: build_initial_monmap 2024-03-29T11:26:15.969 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.977+0000 7fbb0dbe2640 10 monclient: monmap: 2024-03-29T11:26:15.969 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:26:15.969 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:26:15.970 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T11:26:15.970 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T11:26:15.970 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:15.970 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:26:15.970 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:26:15.970 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T11:26:15.970 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T11:26:15.970 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:26:15.970 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.979+0000 7fbb0dbe2640 1 librados: starting msgr at 2024-03-29T11:26:15.970 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.979+0000 7fbb0dbe2640 1 librados: starting objecter 2024-03-29T11:26:15.970 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.979+0000 7fbb0dbe2640 1 Processor -- start 2024-03-29T11:26:15.972 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.980+0000 7fbb0dbe2640 1 -- start start 2024-03-29T11:26:15.972 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.980+0000 7fbb0dbe2640 1 librados: setting wanted keys 2024-03-29T11:26:15.972 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.980+0000 7fbb0dbe2640 1 librados: calling monclient init 2024-03-29T11:26:15.972 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.980+0000 7fbb0dbe2640 10 monclient: init 2024-03-29T11:26:15.972 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.980+0000 7fbb0dbe2640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:15.972 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.980+0000 7fbb0dbe2640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:26:15.972 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.980+0000 7fbb0dbe2640 10 monclient(hunting): picked mon.noname-b con 0x7fbb080a9930 addr v1:172.21.15.176:6789/0 2024-03-29T11:26:15.972 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.980+0000 7fbb0dbe2640 10 monclient(hunting): picked mon.noname-a con 0x7fbb08149dc0 addr v1:172.21.15.114:6789/0 2024-03-29T11:26:15.973 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.980+0000 7fbb0dbe2640 10 monclient(hunting): picked mon.noname-c con 0x7fbb0814ff40 addr v1:172.21.15.114:6790/0 2024-03-29T11:26:15.973 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.980+0000 7fbb0dbe2640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb0813ba20 con 0x7fbb08149dc0 2024-03-29T11:26:15.973 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.980+0000 7fbb0dbe2640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb0813ceb0 con 0x7fbb080a9930 2024-03-29T11:26:15.973 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.980+0000 7fbb0dbe2640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb0813d7e0 con 0x7fbb0814ff40 2024-03-29T11:26:15.973 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.980+0000 7fbb0dbe2640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:15.973 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.980+0000 7fbb0dbe2640 10 monclient(hunting): authenticate will time out at 1246.162970s 2024-03-29T11:26:15.973 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.987+0000 7fbb07fff640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7fbb0814ff40 0x7fbb080ac000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:50051/0 (socket says 172.21.15.114:50051) 2024-03-29T11:26:15.974 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.987+0000 7fbb07fff640 1 -- 172.21.15.114:0/3113128377 learned_addr learned my addr 172.21.15.114:0/3113128377 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:26:15.976 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.989+0000 7fbaeffff640 10 client.?.objecter ms_handle_connect 0x7fbb0814ff40 2024-03-29T11:26:15.976 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.989+0000 7fbaeffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:15.976 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.989+0000 7fbaeffff640 10 client.?.objecter ms_handle_connect 0x7fbb080a9930 2024-03-29T11:26:15.976 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.989+0000 7fbaeffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:15.976 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.989+0000 7fbaeffff640 10 client.?.objecter ms_handle_connect 0x7fbb08149dc0 2024-03-29T11:26:15.976 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.989+0000 7fbaeffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:15.977 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.989+0000 7fbaeffff640 1 -- 172.21.15.114:0/3113128377 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3874809028 0 0) 0x7fbb0813d7e0 con 0x7fbb0814ff40 2024-03-29T11:26:15.977 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.989+0000 7fbaeffff640 10 monclient(hunting): _negotiate 2024-03-29T11:26:15.977 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.989+0000 7fbaeffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:15.977 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.989+0000 7fbaeffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:15.977 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.989+0000 7fbaeffff640 10 monclient(hunting): my global_id is 4436 2024-03-29T11:26:15.977 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.989+0000 7fbaeffff640 1 -- 172.21.15.114:0/3113128377 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbaf4004eb0 con 0x7fbb0814ff40 2024-03-29T11:26:15.977 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.989+0000 7fbaeffff640 1 -- 172.21.15.114:0/3113128377 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 738213074 0 0) 0x7fbb0813ceb0 con 0x7fbb080a9930 2024-03-29T11:26:15.977 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.989+0000 7fbaeffff640 10 monclient(hunting): _negotiate 2024-03-29T11:26:15.977 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.989+0000 7fbaeffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:15.977 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.989+0000 7fbaeffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:15.977 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.989+0000 7fbaeffff640 10 monclient(hunting): my global_id is 4408 2024-03-29T11:26:15.977 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.989+0000 7fbaeffff640 1 -- 172.21.15.114:0/3113128377 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb0813d7e0 con 0x7fbb080a9930 2024-03-29T11:26:15.977 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.989+0000 7fbaeffff640 1 -- 172.21.15.114:0/3113128377 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3184249456 0 0) 0x7fbb0813ba20 con 0x7fbb08149dc0 2024-03-29T11:26:15.983 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.989+0000 7fbaeffff640 10 monclient(hunting): _negotiate 2024-03-29T11:26:15.983 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.989+0000 7fbaeffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:15.984 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.989+0000 7fbaeffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:15.984 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.989+0000 7fbaeffff640 10 monclient(hunting): my global_id is 4422 2024-03-29T11:26:15.984 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.989+0000 7fbaeffff640 1 -- 172.21.15.114:0/3113128377 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb0813ceb0 con 0x7fbb08149dc0 2024-03-29T11:26:15.984 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.989+0000 7fbaeffff640 1 -- 172.21.15.114:0/3113128377 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3166778282 0 0) 0x7fbaf4004eb0 con 0x7fbb0814ff40 2024-03-29T11:26:15.984 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.990+0000 7fbaeffff640 1 -- 172.21.15.114:0/3113128377 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbb0813ba20 con 0x7fbb0814ff40 2024-03-29T11:26:15.984 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.990+0000 7fbaeffff640 1 -- 172.21.15.114:0/3113128377 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3596015780 0 0) 0x7fbb0813d7e0 con 0x7fbb080a9930 2024-03-29T11:26:15.984 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.990+0000 7fbaeffff640 1 -- 172.21.15.114:0/3113128377 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbaf4004eb0 con 0x7fbb080a9930 2024-03-29T11:26:15.984 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.990+0000 7fbaeffff640 1 -- 172.21.15.114:0/3113128377 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 797202745 0 0) 0x7fbb0813ceb0 con 0x7fbb08149dc0 2024-03-29T11:26:15.984 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.990+0000 7fbaeffff640 1 -- 172.21.15.114:0/3113128377 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbb0813d7e0 con 0x7fbb08149dc0 2024-03-29T11:26:15.984 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.990+0000 7fbaeffff640 1 -- 172.21.15.114:0/3113128377 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fbaf8001b70 con 0x7fbb0814ff40 2024-03-29T11:26:15.984 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.990+0000 7fbaeffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:15.984 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.990+0000 7fbaeffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:26:15.984 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.990+0000 7fbaeffff640 10 monclient(hunting): dump: 2024-03-29T11:26:15.984 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:26:15.985 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:26:15.985 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:26:15.985 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:26:15.985 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:15.985 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:26:15.985 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:26:15.985 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:26:15.985 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:26:15.985 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:26:15.985 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.990+0000 7fbaeffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:26:15.985 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.990+0000 7fbaeffff640 1 -- 172.21.15.114:0/3113128377 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fbb00001b20 con 0x7fbb080a9930 2024-03-29T11:26:15.985 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.990+0000 7fbaeffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:15.985 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.990+0000 7fbaeffff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:26:15.985 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.990+0000 7fbaeffff640 10 monclient(hunting): dump: 2024-03-29T11:26:15.985 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:26:15.986 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:26:15.986 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:26:15.986 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:26:15.987 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:15.987 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:26:15.987 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:26:15.987 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:26:15.987 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:26:15.987 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:26:15.987 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.990+0000 7fbaeffff640 1 -- 172.21.15.114:0/3113128377 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fbafc001660 con 0x7fbb08149dc0 2024-03-29T11:26:15.987 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.990+0000 7fbaeffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:15.987 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.990+0000 7fbaeffff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:26:15.987 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.990+0000 7fbaeffff640 10 monclient(hunting): dump: 2024-03-29T11:26:15.987 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:26:15.987 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:26:15.987 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:26:15.987 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:26:15.987 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:15.988 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:26:15.988 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:26:15.988 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:26:15.988 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:26:15.988 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:26:15.988 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.992+0000 7fbaeffff640 1 -- 172.21.15.114:0/3113128377 <== mon.1 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3637813840 0 0) 0x7fbaf4004eb0 con 0x7fbb080a9930 2024-03-29T11:26:15.988 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.992+0000 7fbaeffff640 1 -- 172.21.15.114:0/3113128377 >> v1:172.21.15.114:6790/0 conn(0x7fbb0814ff40 legacy=0x7fbb080ac000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:15.988 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.992+0000 7fbaeffff640 1 -- 172.21.15.114:0/3113128377 >> v1:172.21.15.114:6789/0 conn(0x7fbb08149dc0 legacy=0x7fbb080ab8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:15.988 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.992+0000 7fbaeffff640 10 monclient: _finish_hunting 0 2024-03-29T11:26:15.988 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.992+0000 7fbaeffff640 1 monclient: found mon.b 2024-03-29T11:26:15.988 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.992+0000 7fbaeffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:15.988 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.992+0000 7fbaeffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:26:15.988 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.992+0000 7fbaeffff640 1 -- 172.21.15.114:0/3113128377 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb0813bc70 con 0x7fbb080a9930 2024-03-29T11:26:15.988 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.992+0000 7fbaeffff640 10 monclient: _finish_auth 0 2024-03-29T11:26:15.989 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.992+0000 7fbaeffff640 10 monclient: _check_auth_tickets 2024-03-29T11:26:15.989 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.992+0000 7fbaeffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:15.989 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.992+0000 7fbb0dbe2640 5 monclient: authenticate success, global_id 4408 2024-03-29T11:26:15.989 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.992+0000 7fbb0dbe2640 10 monclient: _renew_subs 2024-03-29T11:26:15.989 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.992+0000 7fbb0dbe2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:26:15.989 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.992+0000 7fbb0dbe2640 1 -- 172.21.15.114:0/3113128377 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb0813dac0 con 0x7fbb080a9930 2024-03-29T11:26:15.989 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.992+0000 7fbb0dbe2640 10 client.4408.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:26:15.989 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.992+0000 7fbb0dbe2640 10 monclient: _renew_subs 2024-03-29T11:26:15.989 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.992+0000 7fbb0dbe2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:26:15.989 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.992+0000 7fbb0dbe2640 1 -- 172.21.15.114:0/3113128377 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb081ef990 con 0x7fbb080a9930 2024-03-29T11:26:15.989 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.992+0000 7fbb0dbe2640 1 librados: init done 2024-03-29T11:26:15.989 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.993+0000 7fbb0dbe2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:15.989 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.993+0000 7fbb0dbe2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:15.989 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.993+0000 7fbb0dbe2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:26:15.989 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.993+0000 7fbb0dbe2640 1 -- 172.21.15.114:0/3113128377 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb08075f10 con 0x7fbb080a9930 2024-03-29T11:26:15.990 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.997+0000 7fbaeffff640 1 -- 172.21.15.114:0/3113128377 <== mon.1 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbb00001ff0 con 0x7fbb080a9930 2024-03-29T11:26:15.990 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.997+0000 7fbaeffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:15.990 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.997+0000 7fbaeffff640 1 -- 172.21.15.114:0/3113128377 <== mon.1 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fbb00004d00 con 0x7fbb080a9930 2024-03-29T11:26:15.990 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.997+0000 7fbaeffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:15.990 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.997+0000 7fbaeffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:26:15.990 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.997+0000 7fbaeffff640 10 monclient: dump: 2024-03-29T11:26:15.990 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:26:15.990 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:26:15.990 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:26:15.990 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:26:15.990 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:15.990 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:26:15.990 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:26:15.990 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:26:15.990 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:26:15.991 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:26:15.991 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:15.997+0000 7fbb057fa640 4 set_mon_vals no callback set 2024-03-29T11:26:15.991 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:16.001+0000 7fbaeffff640 1 -- 172.21.15.114:0/3113128377 <== mon.1 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1594292461 0 0) 0x7fbb00004ee0 con 0x7fbb080a9930 2024-03-29T11:26:15.991 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:16.002+0000 7fbaeffff640 1 -- 172.21.15.114:0/3113128377 <== mon.1 v1:172.21.15.176:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3238987313 0 0) 0x7fbb00084a30 con 0x7fbb080a9930 2024-03-29T11:26:15.992 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:16.002+0000 7fbaeffff640 10 client.4408.objecter ms_dispatch 0x7fbb08000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:26:15.992 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:16.002+0000 7fbaeffff640 3 client.4408.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:26:15.992 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:16.002+0000 7fbaeffff640 3 client.4408.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:26:15.992 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:16.002+0000 7fbaeffff640 20 client.4408.objecter dump_active .. 0 homeless 2024-03-29T11:26:15.992 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:16.002+0000 7fbb06ffd640 10 monclient: get_auth_request con 0x7fbaf406e320 auth_method 0 2024-03-29T11:26:15.992 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:16.002+0000 7fbaeffff640 1 -- 172.21.15.114:0/3113128377 <== mon.1 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbb000b18b0 con 0x7fbb080a9930 2024-03-29T11:26:15.992 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:16.002+0000 7fbaeffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:15.992 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:16.002+0000 7fbaeffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:26:15.992 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:16.003+0000 7fbaeffff640 10 client.4408.objecter ms_handle_connect 0x7fbaf406e320 2024-03-29T11:26:16.315 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:16.329+0000 7fbb0dbe2640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:16.315 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:16.329+0000 7fbb0dbe2640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:16.315 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:16.329+0000 7fbb0dbe2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:26:16.315 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:16.329+0000 7fbb0dbe2640 1 -- 172.21.15.114:0/3113128377 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbb0806ab70 con 0x7fbb080a9930 2024-03-29T11:26:16.318 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:16.332+0000 7fbaeffff640 1 -- 172.21.15.114:0/3113128377 <== mon.1 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2858224613) 0x7fbb00050310 con 0x7fbb080a9930 2024-03-29T11:26:16.318 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:16.332+0000 7fbaeffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:16.318 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:16.332+0000 7fbaeffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:26:16.318 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T11:26:16.318 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"b0de2b9b-c45a-44c7-b6ee-e83b39a1e108","created":"2024-03-29T11:20:49.420660+0000","modified":"2024-03-29T11:20:51.700167+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:51.700167+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a64785d3-6b11-499a-b706-cc827ab96a74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5882dadb-e7a8-4604-ba9a-418013c106c8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d7e8c409-bc66-4e4a-bbdb-420bad31388e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0682989-3595-4cd6-90a5-ab9111c278e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"67068120-e3ff-4817-bd06-44feb7479a35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09e8b-8a85-4997-bdea-75d754e6ee67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2f0d9ead-f47f-4cb3-92a4-0a5ad23cd831","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fdb6a25f-6bd8-4c65-9b9b-b460ba7673a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:26:16.321 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:16.336+0000 7fbb0dbe2640 10 librados: watch_flush enter 2024-03-29T11:26:16.323 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:16.337+0000 7fbb0dbe2640 10 librados: watch_flush exit 2024-03-29T11:26:16.323 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:16.337+0000 7fbb0dbe2640 20 client.4408.objecter shutdown clearing up homeless session... 2024-03-29T11:26:16.323 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:16.337+0000 7fbb0dbe2640 10 client.4408.objecter successfully canceled tick 2024-03-29T11:26:16.324 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:16.338+0000 7fbb0dbe2640 1 -- 172.21.15.114:0/3113128377 >> v1:172.21.15.114:6816/4038755055 conn(0x7fbaf406e320 legacy=0x7fbaf40707e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:16.324 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:16.338+0000 7fbb0dbe2640 10 monclient: shutdown 2024-03-29T11:26:16.324 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:16.338+0000 7fbb0dbe2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:16.324 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:16.338+0000 7fbb0dbe2640 1 -- 172.21.15.114:0/3113128377 >> v1:172.21.15.176:6789/0 conn(0x7fbb080a9930 legacy=0x7fbb0813c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:16.324 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:16.338+0000 7fbb0dbe2640 1 -- 172.21.15.114:0/3113128377 shutdown_connections 2024-03-29T11:26:16.324 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:16.338+0000 7fbb0dbe2640 1 -- 172.21.15.114:0/3113128377 >> 172.21.15.114:0/3113128377 conn(0x7fbb080b35e0 msgr2=0x7fbb080b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:16.324 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:16.338+0000 7fbb0dbe2640 1 -- 172.21.15.114:0/3113128377 shutdown_connections 2024-03-29T11:26:16.325 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:16.338+0000 7fbb0dbe2640 1 -- 172.21.15.114:0/3113128377 wait complete. 2024-03-29T11:26:16.325 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:16.338+0000 7fbb0dbe2640 1 librados: shutdown 2024-03-29T11:26:16.341 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:26:23.148 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:26:23.307 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.321+0000 7f9d90664640 10 monclient: get_monmap_and_config 2024-03-29T11:26:23.308 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.321+0000 7f9d90664640 10 monclient: build_initial_monmap 2024-03-29T11:26:23.308 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.321+0000 7f9d90664640 10 monclient: monmap: 2024-03-29T11:26:23.308 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:26:23.308 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:26:23.308 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:26:23.322719+0000 2024-03-29T11:26:23.308 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:26:23.322719+0000 2024-03-29T11:26:23.308 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:23.308 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:26:23.308 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:26:23.308 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T11:26:23.308 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T11:26:23.308 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:26:23.309 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.323+0000 7f9d90664640 1 Processor -- start 2024-03-29T11:26:23.309 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.323+0000 7f9d90664640 1 -- start start 2024-03-29T11:26:23.309 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.323+0000 7f9d90664640 10 monclient: init 2024-03-29T11:26:23.311 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.324+0000 7f9d90664640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:23.311 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.324+0000 7f9d90664640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:26:23.311 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.324+0000 7f9d90664640 10 monclient(hunting): picked mon.noname-c con 0x7f9d880a9750 addr v1:172.21.15.176:6789/0 2024-03-29T11:26:23.311 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.324+0000 7f9d90664640 10 monclient(hunting): picked mon.noname-a con 0x7f9d8813e1e0 addr v1:172.21.15.114:6789/0 2024-03-29T11:26:23.311 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.324+0000 7f9d8e400640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f9d880a9750 0x7f9d8813dad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.114:56900/0 (socket says 172.21.15.114:56900) 2024-03-29T11:26:23.311 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.324+0000 7f9d90664640 10 monclient(hunting): picked mon.noname-b con 0x7f9d88071c30 addr v1:172.21.15.114:6790/0 2024-03-29T11:26:23.311 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.324+0000 7f9d8e400640 1 -- 172.21.15.114:0/2066974605 learned_addr learned my addr 172.21.15.114:0/2066974605 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:26:23.311 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.325+0000 7f9d90664640 1 -- 172.21.15.114:0/2066974605 --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d880ab300 con 0x7f9d8813e1e0 2024-03-29T11:26:23.311 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.325+0000 7f9d90664640 1 -- 172.21.15.114:0/2066974605 --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d880ac2f0 con 0x7f9d880a9750 2024-03-29T11:26:23.311 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.325+0000 7f9d90664640 1 -- 172.21.15.114:0/2066974605 --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d880ad8d0 con 0x7f9d88071c30 2024-03-29T11:26:23.311 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.325+0000 7f9d90664640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:23.311 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.325+0000 7f9d90664640 10 monclient(hunting): authenticate will time out at 1253.507208s 2024-03-29T11:26:23.314 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.325+0000 7f9d8d3fe640 1 -- 172.21.15.114:0/2066974605 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3050521761 0 0) 0x7f9d880ac2f0 con 0x7f9d880a9750 2024-03-29T11:26:23.315 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.325+0000 7f9d8d3fe640 10 monclient(hunting): _negotiate 2024-03-29T11:26:23.315 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.325+0000 7f9d8d3fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:23.315 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.325+0000 7f9d8d3fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:23.315 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.325+0000 7f9d8d3fe640 10 monclient(hunting): my global_id is 4411 2024-03-29T11:26:23.315 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.325+0000 7f9d8d3fe640 1 -- 172.21.15.114:0/2066974605 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d74003520 con 0x7f9d880a9750 2024-03-29T11:26:23.315 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.326+0000 7f9d8d3fe640 1 -- 172.21.15.114:0/2066974605 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3904824246 0 0) 0x7f9d880ab300 con 0x7f9d8813e1e0 2024-03-29T11:26:23.315 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.326+0000 7f9d8d3fe640 10 monclient(hunting): _negotiate 2024-03-29T11:26:23.315 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.326+0000 7f9d8d3fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:23.315 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.326+0000 7f9d8d3fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:23.315 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.326+0000 7f9d8d3fe640 10 monclient(hunting): my global_id is 4425 2024-03-29T11:26:23.315 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.326+0000 7f9d8d3fe640 1 -- 172.21.15.114:0/2066974605 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d880ac2f0 con 0x7f9d8813e1e0 2024-03-29T11:26:23.315 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.326+0000 7f9d8d3fe640 1 -- 172.21.15.114:0/2066974605 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4250630059 0 0) 0x7f9d74003520 con 0x7f9d880a9750 2024-03-29T11:26:23.315 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.326+0000 7f9d8d3fe640 1 -- 172.21.15.114:0/2066974605 >> v1:172.21.15.114:6790/0 conn(0x7f9d88071c30 legacy=0x7f9d880ace20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:26:23.316 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.326+0000 7f9d8d3fe640 1 -- 172.21.15.114:0/2066974605 >> v1:172.21.15.114:6789/0 conn(0x7f9d8813e1e0 legacy=0x7f9d880ac710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:23.316 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.326+0000 7f9d8d3fe640 10 monclient: _finish_hunting 0 2024-03-29T11:26:23.316 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.326+0000 7f9d8d3fe640 1 monclient: found mon.noname-c 2024-03-29T11:26:23.316 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.326+0000 7f9d8d3fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:23.316 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.326+0000 7f9d8d3fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.176:6789/0 2024-03-29T11:26:23.316 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.326+0000 7f9d8d3fe640 1 -- 172.21.15.114:0/2066974605 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d880ada60 con 0x7f9d880a9750 2024-03-29T11:26:23.316 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.326+0000 7f9d8d3fe640 10 monclient: _finish_auth 0 2024-03-29T11:26:23.316 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.326+0000 7f9d8d3fe640 10 monclient: _check_auth_tickets 2024-03-29T11:26:23.316 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.326+0000 7f9d8d3fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:23.316 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.326+0000 7f9d8d3fe640 1 -- 172.21.15.114:0/2066974605 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f9d84001690 con 0x7f9d880a9750 2024-03-29T11:26:23.316 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.326+0000 7f9d8d3fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:23.316 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.326+0000 7f9d8d3fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:26:23.316 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.326+0000 7f9d8d3fe640 10 monclient: dump: 2024-03-29T11:26:23.316 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:26:23.316 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:26:23.317 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:26:23.317 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:26:23.317 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:23.317 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:26:23.317 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:26:23.317 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:26:23.317 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:26:23.317 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:26:23.317 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.326+0000 7f9d90664640 5 monclient: authenticate success, global_id 4411 2024-03-29T11:26:23.317 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.326+0000 7f9d90664640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:23.317 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.326+0000 7f9d8d3fe640 1 -- 172.21.15.114:0/2066974605 <== mon.1 v1:172.21.15.176:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9d840017f0 con 0x7f9d880a9750 2024-03-29T11:26:23.317 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.326+0000 7f9d8d3fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:23.318 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.326+0000 7f9d8d3fe640 1 -- 172.21.15.114:0/2066974605 <== mon.1 v1:172.21.15.176:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f9d84001c00 con 0x7f9d880a9750 2024-03-29T11:26:23.318 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.326+0000 7f9d8d3fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:23.318 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.326+0000 7f9d8d3fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:26:23.318 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.326+0000 7f9d8d3fe640 10 monclient: dump: 2024-03-29T11:26:23.318 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:26:23.318 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:26:23.318 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:26:23.318 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:26:23.318 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:23.318 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:26:23.318 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:26:23.318 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:26:23.318 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:26:23.318 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:26:23.319 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.327+0000 7f9d90664640 10 monclient: get_monmap_and_config success 2024-03-29T11:26:23.319 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.327+0000 7f9d90664640 4 set_mon_vals no callback set 2024-03-29T11:26:23.319 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.328+0000 7f9d90664640 10 monclient: shutdown 2024-03-29T11:26:23.319 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.328+0000 7f9d90664640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:23.319 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.328+0000 7f9d90664640 1 -- 172.21.15.114:0/2066974605 >> v1:172.21.15.176:6789/0 conn(0x7f9d880a9750 legacy=0x7f9d8813dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:23.319 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.328+0000 7f9d90664640 1 -- 172.21.15.114:0/2066974605 shutdown_connections 2024-03-29T11:26:23.319 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.328+0000 7f9d90664640 1 -- 172.21.15.114:0/2066974605 >> 172.21.15.114:0/2066974605 conn(0x7f9d8806b660 msgr2=0x7f9d8806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:23.319 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.328+0000 7f9d90664640 1 -- 172.21.15.114:0/2066974605 shutdown_connections 2024-03-29T11:26:23.319 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.328+0000 7f9d90664640 1 -- 172.21.15.114:0/2066974605 wait complete. 2024-03-29T11:26:23.327 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.329+0000 7f9d90664640 10 monclient: build_initial_monmap 2024-03-29T11:26:23.327 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.329+0000 7f9d90664640 10 monclient: monmap: 2024-03-29T11:26:23.327 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:26:23.327 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:26:23.327 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T11:26:23.327 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T11:26:23.327 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:23.327 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:26:23.327 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:26:23.327 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T11:26:23.327 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T11:26:23.328 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:26:23.328 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.330+0000 7f9d90664640 1 librados: starting msgr at 2024-03-29T11:26:23.328 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.330+0000 7f9d90664640 1 librados: starting objecter 2024-03-29T11:26:23.328 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.330+0000 7f9d90664640 1 Processor -- start 2024-03-29T11:26:23.328 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.330+0000 7f9d90664640 1 -- start start 2024-03-29T11:26:23.328 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.330+0000 7f9d90664640 1 librados: setting wanted keys 2024-03-29T11:26:23.328 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.330+0000 7f9d90664640 1 librados: calling monclient init 2024-03-29T11:26:23.328 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.330+0000 7f9d90664640 10 monclient: init 2024-03-29T11:26:23.329 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.331+0000 7f9d90664640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:23.329 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.332+0000 7f9d90664640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:26:23.329 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.332+0000 7f9d90664640 10 monclient(hunting): picked mon.noname-b con 0x7f9d88071c30 addr v1:172.21.15.176:6789/0 2024-03-29T11:26:23.329 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.332+0000 7f9d8e400640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f9d88071c30 0x7f9d88141510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.114:57772/0 (socket says 172.21.15.114:57772) 2024-03-29T11:26:23.330 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.332+0000 7f9d90664640 10 monclient(hunting): picked mon.noname-a con 0x7f9d880a9750 addr v1:172.21.15.114:6789/0 2024-03-29T11:26:23.330 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.332+0000 7f9d90664640 10 monclient(hunting): picked mon.noname-c con 0x7f9d8813e1e0 addr v1:172.21.15.114:6790/0 2024-03-29T11:26:23.330 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.332+0000 7f9d90664640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d881381a0 con 0x7f9d880a9750 2024-03-29T11:26:23.330 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.332+0000 7f9d90664640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d881d3020 con 0x7f9d88071c30 2024-03-29T11:26:23.330 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.332+0000 7f9d90664640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d881d31b0 con 0x7f9d8813e1e0 2024-03-29T11:26:23.330 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.332+0000 7f9d90664640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:23.330 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.332+0000 7f9d90664640 10 monclient(hunting): authenticate will time out at 1253.514489s 2024-03-29T11:26:23.330 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.332+0000 7f9d8e400640 1 -- 172.21.15.114:0/146711783 learned_addr learned my addr 172.21.15.114:0/146711783 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:26:23.330 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.332+0000 7f9d7effd640 10 client.?.objecter ms_handle_connect 0x7f9d88071c30 2024-03-29T11:26:23.331 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.332+0000 7f9d7effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:23.331 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.332+0000 7f9d7effd640 10 client.?.objecter ms_handle_connect 0x7f9d880a9750 2024-03-29T11:26:23.331 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.332+0000 7f9d7effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:23.331 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.332+0000 7f9d7effd640 10 client.?.objecter ms_handle_connect 0x7f9d8813e1e0 2024-03-29T11:26:23.331 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.332+0000 7f9d7effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:23.331 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.333+0000 7f9d7effd640 1 -- 172.21.15.114:0/146711783 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1553384710 0 0) 0x7f9d881d3020 con 0x7f9d88071c30 2024-03-29T11:26:23.331 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.333+0000 7f9d7effd640 10 monclient(hunting): _negotiate 2024-03-29T11:26:23.331 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.333+0000 7f9d7effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:23.333 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.333+0000 7f9d7effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:23.334 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.333+0000 7f9d7effd640 10 monclient(hunting): my global_id is 4414 2024-03-29T11:26:23.334 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.333+0000 7f9d7effd640 1 -- 172.21.15.114:0/146711783 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d6c004eb0 con 0x7f9d88071c30 2024-03-29T11:26:23.334 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.333+0000 7f9d7effd640 1 -- 172.21.15.114:0/146711783 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2573796253 0 0) 0x7f9d881d31b0 con 0x7f9d8813e1e0 2024-03-29T11:26:23.334 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.333+0000 7f9d7effd640 10 monclient(hunting): _negotiate 2024-03-29T11:26:23.334 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.333+0000 7f9d7effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:23.334 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.333+0000 7f9d7effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:23.334 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.333+0000 7f9d7effd640 10 monclient(hunting): my global_id is 4439 2024-03-29T11:26:23.334 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.333+0000 7f9d7effd640 1 -- 172.21.15.114:0/146711783 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d881d3020 con 0x7f9d8813e1e0 2024-03-29T11:26:23.334 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.333+0000 7f9d7effd640 1 -- 172.21.15.114:0/146711783 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 959214058 0 0) 0x7f9d6c004eb0 con 0x7f9d88071c30 2024-03-29T11:26:23.334 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.333+0000 7f9d7effd640 1 -- 172.21.15.114:0/146711783 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9d881d31b0 con 0x7f9d88071c30 2024-03-29T11:26:23.334 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.333+0000 7f9d7effd640 1 -- 172.21.15.114:0/146711783 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2842121415 0 0) 0x7f9d881381a0 con 0x7f9d880a9750 2024-03-29T11:26:23.334 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.333+0000 7f9d7effd640 10 monclient(hunting): _negotiate 2024-03-29T11:26:23.334 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.334+0000 7f9d7effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:23.335 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.334+0000 7f9d7effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:23.335 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.334+0000 7f9d7effd640 10 monclient(hunting): my global_id is 4428 2024-03-29T11:26:23.335 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.334+0000 7f9d7effd640 1 -- 172.21.15.114:0/146711783 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d6c004eb0 con 0x7f9d880a9750 2024-03-29T11:26:23.335 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.334+0000 7f9d7effd640 1 -- 172.21.15.114:0/146711783 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f9d84001c00 con 0x7f9d88071c30 2024-03-29T11:26:23.335 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.334+0000 7f9d7effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:23.335 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.334+0000 7f9d7effd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:26:23.335 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.334+0000 7f9d7effd640 10 monclient(hunting): dump: 2024-03-29T11:26:23.335 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:26:23.336 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:26:23.336 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:26:23.336 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:26:23.336 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:23.336 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:26:23.336 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:26:23.336 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:26:23.336 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:26:23.337 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:26:23.337 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.334+0000 7f9d7effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:26:23.337 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.334+0000 7f9d7effd640 1 -- 172.21.15.114:0/146711783 <== mon.1 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1530816441 0 0) 0x7f9d881d31b0 con 0x7f9d88071c30 2024-03-29T11:26:23.337 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.334+0000 7f9d7effd640 1 -- 172.21.15.114:0/146711783 >> v1:172.21.15.114:6790/0 conn(0x7f9d8813e1e0 legacy=0x7f9d88137a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:23.337 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.334+0000 7f9d7effd640 1 -- 172.21.15.114:0/146711783 >> v1:172.21.15.114:6789/0 conn(0x7f9d880a9750 legacy=0x7f9d88137380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:23.337 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.334+0000 7f9d7effd640 10 monclient: _finish_hunting 0 2024-03-29T11:26:23.337 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.334+0000 7f9d7effd640 1 monclient: found mon.b 2024-03-29T11:26:23.337 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.334+0000 7f9d7effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:23.337 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.334+0000 7f9d7effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:26:23.337 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.334+0000 7f9d7effd640 1 -- 172.21.15.114:0/146711783 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d88141c20 con 0x7f9d88071c30 2024-03-29T11:26:23.337 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.334+0000 7f9d7effd640 10 monclient: _finish_auth 0 2024-03-29T11:26:23.337 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.334+0000 7f9d7effd640 10 monclient: _check_auth_tickets 2024-03-29T11:26:23.337 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.334+0000 7f9d7effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:23.337 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.334+0000 7f9d90664640 5 monclient: authenticate success, global_id 4414 2024-03-29T11:26:23.338 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.334+0000 7f9d90664640 10 monclient: _renew_subs 2024-03-29T11:26:23.338 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.334+0000 7f9d90664640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:26:23.338 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.334+0000 7f9d90664640 1 -- 172.21.15.114:0/146711783 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d88141e10 con 0x7f9d88071c30 2024-03-29T11:26:23.338 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.335+0000 7f9d7effd640 1 -- 172.21.15.114:0/146711783 <== mon.1 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9d84004b00 con 0x7f9d88071c30 2024-03-29T11:26:23.338 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.335+0000 7f9d7effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:23.338 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.335+0000 7f9d7effd640 1 -- 172.21.15.114:0/146711783 <== mon.1 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f9d84004f50 con 0x7f9d88071c30 2024-03-29T11:26:23.338 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.335+0000 7f9d7effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:23.338 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.335+0000 7f9d7effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:26:23.339 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.335+0000 7f9d7effd640 10 monclient: dump: 2024-03-29T11:26:23.339 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:26:23.339 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:26:23.339 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:26:23.339 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:26:23.339 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:23.339 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:26:23.339 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:26:23.339 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:26:23.340 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:26:23.340 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:26:23.340 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.335+0000 7f9d90664640 10 client.4414.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:26:23.340 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.335+0000 7f9d90664640 10 monclient: _renew_subs 2024-03-29T11:26:23.340 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.335+0000 7f9d90664640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:26:23.340 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.335+0000 7f9d90664640 1 -- 172.21.15.114:0/146711783 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d88071120 con 0x7f9d88071c30 2024-03-29T11:26:23.340 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.335+0000 7f9d90664640 1 librados: init done 2024-03-29T11:26:23.340 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.335+0000 7f9d8cbfd640 4 set_mon_vals no callback set 2024-03-29T11:26:23.340 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.336+0000 7f9d90664640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:23.340 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.337+0000 7f9d90664640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:23.340 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.337+0000 7f9d90664640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:26:23.340 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.337+0000 7f9d90664640 1 -- 172.21.15.114:0/146711783 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d70005080 con 0x7f9d88071c30 2024-03-29T11:26:23.340 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.338+0000 7f9d7effd640 1 -- 172.21.15.114:0/146711783 <== mon.1 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1594292461 0 0) 0x7f9d8401af20 con 0x7f9d88071c30 2024-03-29T11:26:23.340 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.338+0000 7f9d7effd640 1 -- 172.21.15.114:0/146711783 <== mon.1 v1:172.21.15.176:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3238987313 0 0) 0x7f9d84084b80 con 0x7f9d88071c30 2024-03-29T11:26:23.341 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.338+0000 7f9d7effd640 10 client.4414.objecter ms_dispatch 0x7f9d88000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:26:23.341 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.338+0000 7f9d7effd640 3 client.4414.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:26:23.341 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.338+0000 7f9d7effd640 3 client.4414.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:26:23.341 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.339+0000 7f9d7effd640 20 client.4414.objecter dump_active .. 0 homeless 2024-03-29T11:26:23.341 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.339+0000 7f9d8dbff640 10 monclient: get_auth_request con 0x7f9d6c06dc60 auth_method 0 2024-03-29T11:26:23.341 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.339+0000 7f9d7effd640 10 client.4414.objecter ms_handle_connect 0x7f9d6c06dc60 2024-03-29T11:26:23.342 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.344+0000 7f9d7effd640 1 -- 172.21.15.114:0/146711783 <== mon.1 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9d84050460 con 0x7f9d88071c30 2024-03-29T11:26:23.342 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.344+0000 7f9d7effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:23.342 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.344+0000 7f9d7effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:26:23.720 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.734+0000 7f9d90664640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:23.720 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.734+0000 7f9d90664640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:23.721 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.734+0000 7f9d90664640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:26:23.721 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.734+0000 7f9d90664640 1 -- 172.21.15.114:0/146711783 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9d70005330 con 0x7f9d88071c30 2024-03-29T11:26:23.721 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.735+0000 7f9d7effd640 1 -- 172.21.15.114:0/146711783 <== mon.1 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2858224613) 0x7f9d84052390 con 0x7f9d88071c30 2024-03-29T11:26:23.721 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.735+0000 7f9d7effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:23.722 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.735+0000 7f9d7effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:26:23.722 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T11:26:23.722 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"b0de2b9b-c45a-44c7-b6ee-e83b39a1e108","created":"2024-03-29T11:20:49.420660+0000","modified":"2024-03-29T11:20:51.700167+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:51.700167+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a64785d3-6b11-499a-b706-cc827ab96a74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5882dadb-e7a8-4604-ba9a-418013c106c8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d7e8c409-bc66-4e4a-bbdb-420bad31388e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0682989-3595-4cd6-90a5-ab9111c278e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"67068120-e3ff-4817-bd06-44feb7479a35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09e8b-8a85-4997-bdea-75d754e6ee67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2f0d9ead-f47f-4cb3-92a4-0a5ad23cd831","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fdb6a25f-6bd8-4c65-9b9b-b460ba7673a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:26:23.724 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.739+0000 7f9d90664640 10 librados: watch_flush enter 2024-03-29T11:26:23.725 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.739+0000 7f9d90664640 10 librados: watch_flush exit 2024-03-29T11:26:23.726 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.739+0000 7f9d90664640 20 client.4414.objecter shutdown clearing up homeless session... 2024-03-29T11:26:23.726 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.739+0000 7f9d90664640 10 client.4414.objecter successfully canceled tick 2024-03-29T11:26:23.726 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.739+0000 7f9d90664640 1 -- 172.21.15.114:0/146711783 >> v1:172.21.15.114:6816/4038755055 conn(0x7f9d6c06dc60 legacy=0x7f9d6c070120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:23.726 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.739+0000 7f9d90664640 10 monclient: shutdown 2024-03-29T11:26:23.726 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.739+0000 7f9d90664640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:23.726 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.739+0000 7f9d90664640 1 -- 172.21.15.114:0/146711783 >> v1:172.21.15.176:6789/0 conn(0x7f9d88071c30 legacy=0x7f9d88141510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:23.726 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.739+0000 7f9d90664640 1 -- 172.21.15.114:0/146711783 shutdown_connections 2024-03-29T11:26:23.726 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.739+0000 7f9d90664640 1 -- 172.21.15.114:0/146711783 >> 172.21.15.114:0/146711783 conn(0x7f9d8806b660 msgr2=0x7f9d880ae7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:23.726 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.739+0000 7f9d90664640 1 -- 172.21.15.114:0/146711783 shutdown_connections 2024-03-29T11:26:23.726 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.739+0000 7f9d90664640 1 -- 172.21.15.114:0/146711783 wait complete. 2024-03-29T11:26:23.726 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:23.739+0000 7f9d90664640 1 librados: shutdown 2024-03-29T11:26:23.741 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:26:30.548 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:26:30.680 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.694+0000 7f7508263640 10 monclient: get_monmap_and_config 2024-03-29T11:26:30.681 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.694+0000 7f7508263640 10 monclient: build_initial_monmap 2024-03-29T11:26:30.681 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.694+0000 7f7508263640 10 monclient: monmap: 2024-03-29T11:26:30.681 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:26:30.681 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:26:30.681 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:26:30.695347+0000 2024-03-29T11:26:30.681 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:26:30.695347+0000 2024-03-29T11:26:30.681 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:30.681 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:26:30.681 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:26:30.681 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T11:26:30.681 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T11:26:30.681 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:26:30.682 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.695+0000 7f7508263640 1 Processor -- start 2024-03-29T11:26:30.682 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.695+0000 7f7508263640 1 -- start start 2024-03-29T11:26:30.682 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.695+0000 7f7508263640 10 monclient: init 2024-03-29T11:26:30.683 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.696+0000 7f7508263640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:30.683 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.696+0000 7f7508263640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:26:30.683 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.696+0000 7f7508263640 10 monclient(hunting): picked mon.noname-a con 0x7f75000b0770 addr v1:172.21.15.114:6789/0 2024-03-29T11:26:30.684 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.696+0000 7f7508263640 10 monclient(hunting): picked mon.noname-b con 0x7f750014a700 addr v1:172.21.15.114:6790/0 2024-03-29T11:26:30.684 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.696+0000 7f7508263640 10 monclient(hunting): picked mon.noname-c con 0x7f75000ac5a0 addr v1:172.21.15.176:6789/0 2024-03-29T11:26:30.684 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.696+0000 7f7508263640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75000aba70 con 0x7f75000b0770 2024-03-29T11:26:30.684 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.696+0000 7f7508263640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75000aecb0 con 0x7f75000ac5a0 2024-03-29T11:26:30.684 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.696+0000 7f7508263640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75000af150 con 0x7f750014a700 2024-03-29T11:26:30.684 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.696+0000 7f7508263640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:30.684 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.696+0000 7f7508263640 10 monclient(hunting): authenticate will time out at 1260.879079s 2024-03-29T11:26:30.684 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.697+0000 7f7505fff640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f75000b0770 0x7f7500149ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:35262/0 (socket says 172.21.15.114:35262) 2024-03-29T11:26:30.684 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.697+0000 7f7506800640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f75000ac5a0 0x7f7500151f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.114:36632/0 (socket says 172.21.15.114:36632) 2024-03-29T11:26:30.684 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.697+0000 7f7506800640 1 -- 172.21.15.114:0/2634895150 learned_addr learned my addr 172.21.15.114:0/2634895150 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:26:30.684 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.697+0000 7f7504ffd640 1 -- 172.21.15.114:0/2634895150 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4254466489 0 0) 0x7f75000aba70 con 0x7f75000b0770 2024-03-29T11:26:30.684 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.697+0000 7f7504ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:26:30.684 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.697+0000 7f7504ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:30.684 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.697+0000 7f7504ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:30.685 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.697+0000 7f7504ffd640 10 monclient(hunting): my global_id is 4431 2024-03-29T11:26:30.685 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.697+0000 7f7504ffd640 1 -- 172.21.15.114:0/2634895150 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74ec003520 con 0x7f75000b0770 2024-03-29T11:26:30.685 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.697+0000 7f7504ffd640 1 -- 172.21.15.114:0/2634895150 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 558658231 0 0) 0x7f74ec003520 con 0x7f75000b0770 2024-03-29T11:26:30.685 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.697+0000 7f7504ffd640 1 -- 172.21.15.114:0/2634895150 >> v1:172.21.15.114:6790/0 conn(0x7f750014a700 legacy=0x7f75000abe90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:26:30.686 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.699+0000 7f7504ffd640 1 -- 172.21.15.114:0/2634895150 >> v1:172.21.15.176:6789/0 conn(0x7f75000ac5a0 legacy=0x7f7500151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:30.686 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.699+0000 7f7504ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:26:30.686 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.699+0000 7f7504ffd640 1 monclient: found mon.noname-a 2024-03-29T11:26:30.686 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.699+0000 7f7504ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:30.687 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.699+0000 7f7504ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.114:6789/0 2024-03-29T11:26:30.687 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.699+0000 7f7504ffd640 1 -- 172.21.15.114:0/2634895150 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75000ac9d0 con 0x7f75000b0770 2024-03-29T11:26:30.687 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.699+0000 7f7504ffd640 10 monclient: _finish_auth 0 2024-03-29T11:26:30.687 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.699+0000 7f7504ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:26:30.687 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.699+0000 7f7504ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:30.687 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.699+0000 7f7504ffd640 1 -- 172.21.15.114:0/2634895150 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f74fc0015c0 con 0x7f75000b0770 2024-03-29T11:26:30.687 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.699+0000 7f7504ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:30.687 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.699+0000 7f7504ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:26:30.687 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.699+0000 7f7504ffd640 10 monclient: dump: 2024-03-29T11:26:30.687 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:26:30.688 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:26:30.688 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:26:30.688 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:26:30.688 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:30.688 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:26:30.688 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:26:30.688 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:26:30.688 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:26:30.688 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:26:30.688 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.700+0000 7f7504ffd640 1 -- 172.21.15.114:0/2634895150 <== mon.0 v1:172.21.15.114:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f74fc001890 con 0x7f75000b0770 2024-03-29T11:26:30.688 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.700+0000 7f7504ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:30.688 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.700+0000 7f7504ffd640 1 -- 172.21.15.114:0/2634895150 <== mon.0 v1:172.21.15.114:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f74fc001c20 con 0x7f75000b0770 2024-03-29T11:26:30.688 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.700+0000 7f7504ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:30.691 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.700+0000 7f7504ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:26:30.691 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.700+0000 7f7504ffd640 10 monclient: dump: 2024-03-29T11:26:30.691 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:26:30.692 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:26:30.692 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:26:30.692 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:26:30.692 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:30.692 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:26:30.692 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:26:30.692 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:26:30.692 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:26:30.692 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:26:30.692 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.700+0000 7f7508263640 5 monclient: authenticate success, global_id 4431 2024-03-29T11:26:30.692 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.700+0000 7f7508263640 10 monclient: get_monmap_and_config success 2024-03-29T11:26:30.692 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.700+0000 7f7508263640 4 set_mon_vals no callback set 2024-03-29T11:26:30.693 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.701+0000 7f7508263640 10 monclient: shutdown 2024-03-29T11:26:30.693 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.701+0000 7f7508263640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:30.693 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.701+0000 7f7508263640 1 -- 172.21.15.114:0/2634895150 >> v1:172.21.15.114:6789/0 conn(0x7f75000b0770 legacy=0x7f7500149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:30.693 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.701+0000 7f7508263640 1 -- 172.21.15.114:0/2634895150 shutdown_connections 2024-03-29T11:26:30.693 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.701+0000 7f7508263640 1 -- 172.21.15.114:0/2634895150 >> 172.21.15.114:0/2634895150 conn(0x7f750006b660 msgr2=0x7f750006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:30.693 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.701+0000 7f7508263640 1 -- 172.21.15.114:0/2634895150 shutdown_connections 2024-03-29T11:26:30.693 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.701+0000 7f7508263640 1 -- 172.21.15.114:0/2634895150 wait complete. 2024-03-29T11:26:30.693 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.702+0000 7f7508263640 10 monclient: build_initial_monmap 2024-03-29T11:26:30.693 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.702+0000 7f7508263640 10 monclient: monmap: 2024-03-29T11:26:30.693 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:26:30.693 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:26:30.693 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T11:26:30.694 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T11:26:30.694 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:30.694 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:26:30.694 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:26:30.694 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T11:26:30.694 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T11:26:30.694 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:26:30.694 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.702+0000 7f7508263640 1 librados: starting msgr at 2024-03-29T11:26:30.694 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.702+0000 7f7508263640 1 librados: starting objecter 2024-03-29T11:26:30.694 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.702+0000 7f7508263640 1 Processor -- start 2024-03-29T11:26:30.694 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.702+0000 7f7508263640 1 -- start start 2024-03-29T11:26:30.694 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.702+0000 7f7508263640 1 librados: setting wanted keys 2024-03-29T11:26:30.694 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.702+0000 7f7508263640 1 librados: calling monclient init 2024-03-29T11:26:30.695 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.702+0000 7f7508263640 10 monclient: init 2024-03-29T11:26:30.695 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.703+0000 7f7508263640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:30.695 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.703+0000 7f7508263640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:26:30.695 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.703+0000 7f7508263640 10 monclient(hunting): picked mon.noname-c con 0x7f75000ac5a0 addr v1:172.21.15.114:6790/0 2024-03-29T11:26:30.695 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.703+0000 7f7508263640 10 monclient(hunting): picked mon.noname-b con 0x7f75000b0770 addr v1:172.21.15.176:6789/0 2024-03-29T11:26:30.695 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.704+0000 7f7506800640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f750014a700 0x7f75001e6c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:35508/0 (socket says 172.21.15.114:35508) 2024-03-29T11:26:30.695 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.704+0000 7f7506800640 1 -- 172.21.15.114:0/975434027 learned_addr learned my addr 172.21.15.114:0/975434027 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:26:30.695 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.704+0000 7f7508263640 10 monclient(hunting): picked mon.noname-a con 0x7f750014a700 addr v1:172.21.15.114:6789/0 2024-03-29T11:26:30.695 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.704+0000 7f74f67fc640 10 client.?.objecter ms_handle_connect 0x7f750014a700 2024-03-29T11:26:30.695 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.704+0000 7f74f67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:30.695 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.704+0000 7f7508263640 1 -- 172.21.15.114:0/975434027 --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7500125440 con 0x7f750014a700 2024-03-29T11:26:30.695 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.704+0000 7f7508263640 1 -- 172.21.15.114:0/975434027 --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7500129150 con 0x7f75000b0770 2024-03-29T11:26:30.696 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.704+0000 7f74f67fc640 1 -- 172.21.15.114:0/975434027 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4079487226 0 0) 0x7f7500125440 con 0x7f750014a700 2024-03-29T11:26:30.696 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.704+0000 7f7508263640 1 -- 172.21.15.114:0/975434027 --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75001292e0 con 0x7f75000ac5a0 2024-03-29T11:26:30.696 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.704+0000 7f7508263640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:30.696 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.704+0000 7f7508263640 10 monclient(hunting): authenticate will time out at 1260.886917s 2024-03-29T11:26:30.696 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.704+0000 7f74f67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:26:30.696 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.704+0000 7f74f67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:30.696 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.704+0000 7f74f67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:30.696 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.704+0000 7f74f67fc640 10 monclient(hunting): my global_id is 4434 2024-03-29T11:26:30.696 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.704+0000 7f74f67fc640 1 -- 172.21.15.114:0/975434027 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74e4004eb0 con 0x7f750014a700 2024-03-29T11:26:30.696 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.705+0000 7f74f67fc640 1 -- 172.21.15.114:0/975434027 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2334669680 0 0) 0x7f74e4004eb0 con 0x7f750014a700 2024-03-29T11:26:30.696 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.705+0000 7f74f67fc640 1 -- 172.21.15.114:0/975434027 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7500125440 con 0x7f750014a700 2024-03-29T11:26:30.696 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.705+0000 7f74f67fc640 1 -- 172.21.15.114:0/975434027 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f74f8001c30 con 0x7f750014a700 2024-03-29T11:26:30.696 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.705+0000 7f74f67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:30.697 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.705+0000 7f74f67fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:26:30.697 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.705+0000 7f74f67fc640 10 monclient(hunting): dump: 2024-03-29T11:26:30.697 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:26:30.697 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:26:30.697 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:26:30.697 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:26:30.697 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:30.697 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:26:30.697 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:26:30.697 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:26:30.697 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:26:30.697 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:26:30.697 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.705+0000 7f74f67fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:26:30.697 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.705+0000 7f74f67fc640 1 -- 172.21.15.114:0/975434027 <== mon.0 v1:172.21.15.114:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 332894134 0 0) 0x7f7500125440 con 0x7f750014a700 2024-03-29T11:26:30.698 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.705+0000 7f74f67fc640 1 -- 172.21.15.114:0/975434027 >> v1:172.21.15.114:6790/0 conn(0x7f75000ac5a0 legacy=0x7f7500146e00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:26:30.699 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.705+0000 7f74f67fc640 1 -- 172.21.15.114:0/975434027 >> v1:172.21.15.176:6789/0 conn(0x7f75000b0770 legacy=0x7f75001dec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:30.699 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.705+0000 7f74f67fc640 10 monclient: _finish_hunting 0 2024-03-29T11:26:30.699 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.705+0000 7f74f67fc640 1 monclient: found mon.a 2024-03-29T11:26:30.699 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.705+0000 7f74f67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:30.699 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.705+0000 7f74f67fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T11:26:30.700 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.705+0000 7f74f67fc640 1 -- 172.21.15.114:0/975434027 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7500147aa0 con 0x7f750014a700 2024-03-29T11:26:30.700 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.705+0000 7f74f67fc640 10 monclient: _finish_auth 0 2024-03-29T11:26:30.700 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.705+0000 7f74f67fc640 10 monclient: _check_auth_tickets 2024-03-29T11:26:30.700 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.705+0000 7f74f67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:30.700 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.706+0000 7f7508263640 5 monclient: authenticate success, global_id 4434 2024-03-29T11:26:30.700 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.706+0000 7f7508263640 10 monclient: _renew_subs 2024-03-29T11:26:30.700 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.706+0000 7f7508263640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T11:26:30.700 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.706+0000 7f7508263640 1 -- 172.21.15.114:0/975434027 --> v1:172.21.15.114:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7500147cf0 con 0x7f750014a700 2024-03-29T11:26:30.700 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.706+0000 7f7508263640 10 client.4434.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:26:30.700 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.706+0000 7f7508263640 10 monclient: _renew_subs 2024-03-29T11:26:30.700 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.706+0000 7f7508263640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T11:26:30.700 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.706+0000 7f7508263640 1 -- 172.21.15.114:0/975434027 --> v1:172.21.15.114:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7500128d20 con 0x7f750014a700 2024-03-29T11:26:30.700 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.706+0000 7f7508263640 1 librados: init done 2024-03-29T11:26:30.701 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.706+0000 7f74f67fc640 1 -- 172.21.15.114:0/975434027 <== mon.0 v1:172.21.15.114:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f74f8001d90 con 0x7f750014a700 2024-03-29T11:26:30.701 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.706+0000 7f74f67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:30.701 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.706+0000 7f74f67fc640 1 -- 172.21.15.114:0/975434027 <== mon.0 v1:172.21.15.114:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f74f8004e40 con 0x7f750014a700 2024-03-29T11:26:30.701 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.706+0000 7f74f67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:30.701 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.706+0000 7f74f67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:26:30.701 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.706+0000 7f74f67fc640 10 monclient: dump: 2024-03-29T11:26:30.701 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:26:30.701 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:26:30.701 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:26:30.701 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:26:30.701 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:30.701 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:26:30.701 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:26:30.701 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:26:30.701 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:26:30.702 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:26:30.702 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.706+0000 7f74f7fff640 4 set_mon_vals no callback set 2024-03-29T11:26:30.702 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.707+0000 7f74e3fff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:30.702 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.707+0000 7f74e3fff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:30.702 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.707+0000 7f74e3fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T11:26:30.702 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.707+0000 7f74e3fff640 1 -- 172.21.15.114:0/975434027 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7500075f10 con 0x7f750014a700 2024-03-29T11:26:30.702 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.708+0000 7f74f67fc640 1 -- 172.21.15.114:0/975434027 <== mon.0 v1:172.21.15.114:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1594292461 0 0) 0x7f74f801ae10 con 0x7f750014a700 2024-03-29T11:26:30.702 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.709+0000 7f74f67fc640 1 -- 172.21.15.114:0/975434027 <== mon.0 v1:172.21.15.114:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3238987313 0 0) 0x7f74f8084d60 con 0x7f750014a700 2024-03-29T11:26:30.702 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.709+0000 7f74f67fc640 10 client.4434.objecter ms_dispatch 0x7f7500000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:26:30.702 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.709+0000 7f74f67fc640 3 client.4434.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:26:30.702 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.709+0000 7f74f67fc640 3 client.4434.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:26:30.702 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.709+0000 7f74f67fc640 20 client.4434.objecter dump_active .. 0 homeless 2024-03-29T11:26:30.702 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.709+0000 7f7505fff640 10 monclient: get_auth_request con 0x7f74e406dc00 auth_method 0 2024-03-29T11:26:30.702 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.709+0000 7f74f67fc640 10 client.4434.objecter ms_handle_connect 0x7f74e406dc00 2024-03-29T11:26:30.703 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.714+0000 7f74f67fc640 1 -- 172.21.15.114:0/975434027 <== mon.0 v1:172.21.15.114:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f74f8050470 con 0x7f750014a700 2024-03-29T11:26:30.703 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.714+0000 7f74f67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:30.703 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:30.714+0000 7f74f67fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:26:31.007 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:31.020+0000 7f74e3fff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:31.007 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:31.020+0000 7f74e3fff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:31.007 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:31.020+0000 7f74e3fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T11:26:31.007 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:31.020+0000 7f74e3fff640 1 -- 172.21.15.114:0/975434027 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f75000008d0 con 0x7f750014a700 2024-03-29T11:26:31.007 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:31.021+0000 7f74f67fc640 1 -- 172.21.15.114:0/975434027 <== mon.0 v1:172.21.15.114:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2858224613) 0x7f74f8052570 con 0x7f750014a700 2024-03-29T11:26:31.007 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:31.021+0000 7f74f67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:31.007 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:31.021+0000 7f74f67fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:26:31.008 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T11:26:31.008 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"b0de2b9b-c45a-44c7-b6ee-e83b39a1e108","created":"2024-03-29T11:20:49.420660+0000","modified":"2024-03-29T11:20:51.700167+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:51.700167+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a64785d3-6b11-499a-b706-cc827ab96a74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5882dadb-e7a8-4604-ba9a-418013c106c8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d7e8c409-bc66-4e4a-bbdb-420bad31388e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0682989-3595-4cd6-90a5-ab9111c278e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"67068120-e3ff-4817-bd06-44feb7479a35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09e8b-8a85-4997-bdea-75d754e6ee67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2f0d9ead-f47f-4cb3-92a4-0a5ad23cd831","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fdb6a25f-6bd8-4c65-9b9b-b460ba7673a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:26:31.009 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:31.023+0000 7f74e3fff640 10 librados: watch_flush enter 2024-03-29T11:26:31.009 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:31.023+0000 7f74e3fff640 10 librados: watch_flush exit 2024-03-29T11:26:31.009 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:31.023+0000 7f74e3fff640 20 client.4434.objecter shutdown clearing up homeless session... 2024-03-29T11:26:31.009 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:31.023+0000 7f74e3fff640 10 client.4434.objecter successfully canceled tick 2024-03-29T11:26:31.010 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:31.024+0000 7f74e3fff640 1 -- 172.21.15.114:0/975434027 >> v1:172.21.15.114:6816/4038755055 conn(0x7f74e406dc00 legacy=0x7f74e40700c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:31.010 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:31.024+0000 7f74e3fff640 10 monclient: shutdown 2024-03-29T11:26:31.010 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:31.024+0000 7f74e3fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:31.010 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:31.024+0000 7f74e3fff640 1 -- 172.21.15.114:0/975434027 >> v1:172.21.15.114:6789/0 conn(0x7f750014a700 legacy=0x7f75001e6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:31.010 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:31.024+0000 7f74e3fff640 1 -- 172.21.15.114:0/975434027 shutdown_connections 2024-03-29T11:26:31.011 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:31.024+0000 7f74e3fff640 1 -- 172.21.15.114:0/975434027 >> 172.21.15.114:0/975434027 conn(0x7f750006b660 msgr2=0x7f75000712d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:31.011 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:31.025+0000 7f74e3fff640 1 -- 172.21.15.114:0/975434027 shutdown_connections 2024-03-29T11:26:31.011 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:31.025+0000 7f74e3fff640 1 -- 172.21.15.114:0/975434027 wait complete. 2024-03-29T11:26:31.011 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:31.025+0000 7f74e3fff640 1 librados: shutdown 2024-03-29T11:26:31.024 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:26:37.832 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:26:37.983 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:37.996+0000 7f8c1a496640 10 monclient: get_monmap_and_config 2024-03-29T11:26:37.983 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:37.996+0000 7f8c1a496640 10 monclient: build_initial_monmap 2024-03-29T11:26:37.983 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:37.996+0000 7f8c1a496640 10 monclient: monmap: 2024-03-29T11:26:37.983 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:26:37.983 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:26:37.983 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:26:37.997578+0000 2024-03-29T11:26:37.983 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:26:37.997578+0000 2024-03-29T11:26:37.984 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:37.984 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:26:37.984 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:26:37.984 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T11:26:37.984 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T11:26:37.984 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:26:37.985 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:37.998+0000 7f8c1a496640 1 Processor -- start 2024-03-29T11:26:37.985 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:37.998+0000 7f8c1a496640 1 -- start start 2024-03-29T11:26:37.985 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:37.998+0000 7f8c1a496640 10 monclient: init 2024-03-29T11:26:37.986 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:37.999+0000 7f8c1a496640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:37.986 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:37.999+0000 7f8c1a496640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:26:37.987 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:37.999+0000 7f8c1a496640 10 monclient(hunting): picked mon.noname-a con 0x7f8c14071de0 addr v1:172.21.15.114:6789/0 2024-03-29T11:26:37.987 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:37.999+0000 7f8c1a496640 10 monclient(hunting): picked mon.noname-b con 0x7f8c140721c0 addr v1:172.21.15.114:6790/0 2024-03-29T11:26:37.987 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:37.999+0000 7f8c1a496640 10 monclient(hunting): picked mon.noname-c con 0x7f8c14145ae0 addr v1:172.21.15.176:6789/0 2024-03-29T11:26:37.987 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:37.999+0000 7f8c1a496640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c14145650 con 0x7f8c14071de0 2024-03-29T11:26:37.988 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:37.999+0000 7f8c1a496640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c140ae260 con 0x7f8c14145ae0 2024-03-29T11:26:37.988 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:37.999+0000 7f8c1a496640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c141420d0 con 0x7f8c140721c0 2024-03-29T11:26:37.988 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:37.999+0000 7f8c1a496640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:37.988 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:37.999+0000 7f8c1a496640 10 monclient(hunting): authenticate will time out at 1268.181795s 2024-03-29T11:26:37.988 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:37.999+0000 7f8c137fe640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f8c140721c0 0x7f8c1413f400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:39092/0 (socket says 172.21.15.114:39092) 2024-03-29T11:26:37.988 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:37.999+0000 7f8c137fe640 1 -- 172.21.15.114:0/3417894573 learned_addr learned my addr 172.21.15.114:0/3417894573 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:26:37.988 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.001+0000 7f8c12ffd640 1 -- 172.21.15.114:0/3417894573 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2720837062 0 0) 0x7f8c141420d0 con 0x7f8c140721c0 2024-03-29T11:26:37.988 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.001+0000 7f8c12ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:26:37.988 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.001+0000 7f8c12ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:37.988 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.001+0000 7f8c12ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:37.988 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.001+0000 7f8c12ffd640 10 monclient(hunting): my global_id is 4442 2024-03-29T11:26:37.988 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.001+0000 7f8c12ffd640 1 -- 172.21.15.114:0/3417894573 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8bfc003520 con 0x7f8c140721c0 2024-03-29T11:26:37.988 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.001+0000 7f8c12ffd640 1 -- 172.21.15.114:0/3417894573 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1827938735 0 0) 0x7f8bfc003520 con 0x7f8c140721c0 2024-03-29T11:26:37.988 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.001+0000 7f8c12ffd640 1 -- 172.21.15.114:0/3417894573 >> v1:172.21.15.176:6789/0 conn(0x7f8c14145ae0 legacy=0x7f8c14145f50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:26:37.990 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.003+0000 7f8c12ffd640 1 -- 172.21.15.114:0/3417894573 >> v1:172.21.15.114:6789/0 conn(0x7f8c14071de0 legacy=0x7f8c140aed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:37.990 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.003+0000 7f8c12ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:26:37.990 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.003+0000 7f8c12ffd640 1 monclient: found mon.noname-b 2024-03-29T11:26:37.990 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.003+0000 7f8c12ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:37.990 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.003+0000 7f8c12ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.114:6790/0 2024-03-29T11:26:37.990 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.003+0000 7f8c12ffd640 1 -- 172.21.15.114:0/3417894573 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c140b0590 con 0x7f8c140721c0 2024-03-29T11:26:37.990 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.003+0000 7f8c12ffd640 10 monclient: _finish_auth 0 2024-03-29T11:26:37.991 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.003+0000 7f8c12ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:26:37.991 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.003+0000 7f8c12ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:37.991 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.003+0000 7f8c1a496640 5 monclient: authenticate success, global_id 4442 2024-03-29T11:26:37.991 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.003+0000 7f8c1a496640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:37.991 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.003+0000 7f8c12ffd640 1 -- 172.21.15.114:0/3417894573 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f8c04001690 con 0x7f8c140721c0 2024-03-29T11:26:37.991 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.003+0000 7f8c12ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:37.991 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.003+0000 7f8c12ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:26:37.991 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.003+0000 7f8c12ffd640 10 monclient: dump: 2024-03-29T11:26:37.991 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:26:37.991 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:26:37.991 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:26:37.991 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:26:37.991 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:37.991 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:26:37.992 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:26:37.992 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:26:37.992 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:26:37.992 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:26:37.992 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.003+0000 7f8c1a496640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:37.993 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.007+0000 7f8c12ffd640 1 -- 172.21.15.114:0/3417894573 <== mon.2 v1:172.21.15.114:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8c040018b0 con 0x7f8c140721c0 2024-03-29T11:26:37.993 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.007+0000 7f8c12ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:37.993 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.007+0000 7f8c1a496640 10 monclient: get_monmap_and_config success 2024-03-29T11:26:37.993 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.007+0000 7f8c1a496640 4 set_mon_vals no callback set 2024-03-29T11:26:37.994 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.008+0000 7f8c1a496640 10 monclient: shutdown 2024-03-29T11:26:37.995 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.008+0000 7f8c1a496640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:37.995 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.008+0000 7f8c1a496640 1 -- 172.21.15.114:0/3417894573 >> v1:172.21.15.114:6790/0 conn(0x7f8c140721c0 legacy=0x7f8c1413f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:37.995 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.008+0000 7f8c1a496640 1 -- 172.21.15.114:0/3417894573 shutdown_connections 2024-03-29T11:26:37.995 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.008+0000 7f8c1a496640 1 -- 172.21.15.114:0/3417894573 >> 172.21.15.114:0/3417894573 conn(0x7f8c1406b660 msgr2=0x7f8c1406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:37.996 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.009+0000 7f8c1a496640 1 -- 172.21.15.114:0/3417894573 shutdown_connections 2024-03-29T11:26:37.997 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.009+0000 7f8c1a496640 1 -- 172.21.15.114:0/3417894573 wait complete. 2024-03-29T11:26:37.997 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.010+0000 7f8c1a496640 10 monclient: build_initial_monmap 2024-03-29T11:26:37.998 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.010+0000 7f8c1a496640 10 monclient: monmap: 2024-03-29T11:26:37.998 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:26:37.999 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:26:37.999 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T11:26:37.999 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T11:26:37.999 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:37.999 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:26:37.999 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:26:37.999 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T11:26:37.999 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T11:26:38.000 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:26:38.000 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.011+0000 7f8c1a496640 1 librados: starting msgr at 2024-03-29T11:26:38.000 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.011+0000 7f8c1a496640 1 librados: starting objecter 2024-03-29T11:26:38.000 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.011+0000 7f8c1a496640 1 Processor -- start 2024-03-29T11:26:38.000 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.011+0000 7f8c1a496640 1 -- start start 2024-03-29T11:26:38.001 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.011+0000 7f8c1a496640 1 librados: setting wanted keys 2024-03-29T11:26:38.001 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.011+0000 7f8c1a496640 1 librados: calling monclient init 2024-03-29T11:26:38.001 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.011+0000 7f8c1a496640 10 monclient: init 2024-03-29T11:26:38.001 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.012+0000 7f8c1a496640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:38.001 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.012+0000 7f8c1a496640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:26:38.001 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.012+0000 7f8c1a496640 10 monclient(hunting): picked mon.noname-c con 0x7f8c14071de0 addr v1:172.21.15.114:6790/0 2024-03-29T11:26:38.001 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.012+0000 7f8c1a496640 10 monclient(hunting): picked mon.noname-a con 0x7f8c14145ae0 addr v1:172.21.15.114:6789/0 2024-03-29T11:26:38.001 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.012+0000 7f8c1a496640 10 monclient(hunting): picked mon.noname-b con 0x7f8c14135b70 addr v1:172.21.15.176:6789/0 2024-03-29T11:26:38.001 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.012+0000 7f8c1a496640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c14132ff0 con 0x7f8c14145ae0 2024-03-29T11:26:38.001 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.012+0000 7f8c1a496640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c14134070 con 0x7f8c14135b70 2024-03-29T11:26:38.001 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.012+0000 7f8c1a496640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c14132be0 con 0x7f8c14071de0 2024-03-29T11:26:38.001 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.012+0000 7f8c1a496640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:38.001 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.012+0000 7f8c1a496640 10 monclient(hunting): authenticate will time out at 1268.194930s 2024-03-29T11:26:38.001 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.013+0000 7f8c13fff640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f8c14071de0 0x7f8c14131f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:42482/0 (socket says 172.21.15.114:42482) 2024-03-29T11:26:38.001 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.013+0000 7f8c13fff640 1 -- 172.21.15.114:0/3637590515 learned_addr learned my addr 172.21.15.114:0/3637590515 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:26:38.002 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.015+0000 7f8c10ff9640 10 client.?.objecter ms_handle_connect 0x7f8c14071de0 2024-03-29T11:26:38.002 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.015+0000 7f8c10ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:38.002 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.015+0000 7f8c10ff9640 1 -- 172.21.15.114:0/3637590515 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4221427534 0 0) 0x7f8c14132be0 con 0x7f8c14071de0 2024-03-29T11:26:38.002 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.015+0000 7f8c10ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:26:38.002 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.015+0000 7f8c10ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:38.002 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.015+0000 7f8c10ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:38.002 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.015+0000 7f8c10ff9640 10 monclient(hunting): my global_id is 4445 2024-03-29T11:26:38.003 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.015+0000 7f8c10ff9640 1 -- 172.21.15.114:0/3637590515 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c00004eb0 con 0x7f8c14071de0 2024-03-29T11:26:38.003 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.015+0000 7f8c10ff9640 1 -- 172.21.15.114:0/3637590515 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 579700772 0 0) 0x7f8c00004eb0 con 0x7f8c14071de0 2024-03-29T11:26:38.003 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.015+0000 7f8c10ff9640 1 -- 172.21.15.114:0/3637590515 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8c14132be0 con 0x7f8c14071de0 2024-03-29T11:26:38.003 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.015+0000 7f8c10ff9640 1 -- 172.21.15.114:0/3637590515 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f8c0c001c70 con 0x7f8c14071de0 2024-03-29T11:26:38.003 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.015+0000 7f8c10ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:38.003 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.015+0000 7f8c10ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:26:38.003 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.015+0000 7f8c10ff9640 10 monclient(hunting): dump: 2024-03-29T11:26:38.003 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:26:38.003 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:26:38.003 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:26:38.003 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:26:38.003 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:38.004 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:26:38.004 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:26:38.004 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:26:38.004 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:26:38.004 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:26:38.004 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.015+0000 7f8c10ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:26:38.004 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.016+0000 7f8c10ff9640 1 -- 172.21.15.114:0/3637590515 <== mon.2 v1:172.21.15.114:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1278202532 0 0) 0x7f8c14132be0 con 0x7f8c14071de0 2024-03-29T11:26:38.004 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.016+0000 7f8c10ff9640 1 -- 172.21.15.114:0/3637590515 >> v1:172.21.15.176:6789/0 conn(0x7f8c14135b70 legacy=0x7f8c141d7540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:26:38.008 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.018+0000 7f8c10ff9640 1 -- 172.21.15.114:0/3637590515 >> v1:172.21.15.114:6789/0 conn(0x7f8c14145ae0 legacy=0x7f8c141d6e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:38.008 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.018+0000 7f8c10ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:26:38.008 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.018+0000 7f8c10ff9640 1 monclient: found mon.c 2024-03-29T11:26:38.008 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.018+0000 7f8c10ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:38.008 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.018+0000 7f8c10ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:26:38.008 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.018+0000 7f8c10ff9640 1 -- 172.21.15.114:0/3637590515 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c141e5840 con 0x7f8c14071de0 2024-03-29T11:26:38.008 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.019+0000 7f8c10ff9640 10 monclient: _finish_auth 0 2024-03-29T11:26:38.008 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.019+0000 7f8c10ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:26:38.008 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.019+0000 7f8c10ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:38.008 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.019+0000 7f8c10ff9640 10 client.?.objecter ms_handle_connect 0x7f8c14145ae0 2024-03-29T11:26:38.008 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.019+0000 7f8c10ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:38.009 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.019+0000 7f8c1a496640 5 monclient: authenticate success, global_id 4445 2024-03-29T11:26:38.009 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.019+0000 7f8c1a496640 10 monclient: _renew_subs 2024-03-29T11:26:38.009 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.019+0000 7f8c1a496640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:26:38.009 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.019+0000 7f8c1a496640 1 -- 172.21.15.114:0/3637590515 --> v1:172.21.15.114:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c141e2810 con 0x7f8c14071de0 2024-03-29T11:26:38.009 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.019+0000 7f8c1a496640 10 client.4445.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:26:38.009 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.019+0000 7f8c1a496640 10 monclient: _renew_subs 2024-03-29T11:26:38.009 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.019+0000 7f8c1a496640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:26:38.009 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.019+0000 7f8c1a496640 1 -- 172.21.15.114:0/3637590515 --> v1:172.21.15.114:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c141e2d40 con 0x7f8c14071de0 2024-03-29T11:26:38.009 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.019+0000 7f8c1a496640 1 librados: init done 2024-03-29T11:26:38.009 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.019+0000 7f8c10ff9640 1 -- 172.21.15.114:0/3637590515 <== mon.2 v1:172.21.15.114:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8c0c0020c0 con 0x7f8c14071de0 2024-03-29T11:26:38.009 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.019+0000 7f8c10ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:38.009 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.019+0000 7f8c127fc640 4 set_mon_vals no callback set 2024-03-29T11:26:38.009 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.020+0000 7f8c10ff9640 1 -- 172.21.15.114:0/3637590515 <== mon.2 v1:172.21.15.114:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f8c0c003de0 con 0x7f8c14071de0 2024-03-29T11:26:38.009 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.020+0000 7f8c10ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:38.010 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.020+0000 7f8c10ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:38.010 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.020+0000 7f8c10ff9640 10 monclient: dump: 2024-03-29T11:26:38.010 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:26:38.010 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:26:38.010 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:26:38.010 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:26:38.010 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:38.010 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:26:38.010 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:26:38.010 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:26:38.010 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:26:38.010 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:26:38.010 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.021+0000 7f8c1a496640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:38.010 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.021+0000 7f8c1a496640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:38.010 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.021+0000 7f8c1a496640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:26:38.011 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.021+0000 7f8c1a496640 1 -- 172.21.15.114:0/3637590515 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bf4005080 con 0x7f8c14071de0 2024-03-29T11:26:38.012 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.022+0000 7f8c10ff9640 1 -- 172.21.15.114:0/3637590515 <== mon.2 v1:172.21.15.114:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1594292461 0 0) 0x7f8c0c003f40 con 0x7f8c14071de0 2024-03-29T11:26:38.012 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.022+0000 7f8c10ff9640 1 -- 172.21.15.114:0/3637590515 <== mon.2 v1:172.21.15.114:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3238987313 0 0) 0x7f8c0c084e30 con 0x7f8c14071de0 2024-03-29T11:26:38.012 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.022+0000 7f8c10ff9640 10 client.4445.objecter ms_dispatch 0x7f8c14000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:26:38.012 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.022+0000 7f8c10ff9640 3 client.4445.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:26:38.012 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.022+0000 7f8c10ff9640 3 client.4445.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:26:38.012 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.022+0000 7f8c10ff9640 20 client.4445.objecter dump_active .. 0 homeless 2024-03-29T11:26:38.012 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.024+0000 7f8c137fe640 10 monclient: get_auth_request con 0x7f8c0006db80 auth_method 0 2024-03-29T11:26:38.012 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.025+0000 7f8c10ff9640 10 client.4445.objecter ms_handle_connect 0x7f8c0006db80 2024-03-29T11:26:38.019 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.031+0000 7f8c10ff9640 1 -- 172.21.15.114:0/3637590515 <== mon.2 v1:172.21.15.114:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8c0c050710 con 0x7f8c14071de0 2024-03-29T11:26:38.019 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.031+0000 7f8c10ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:38.019 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.031+0000 7f8c10ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:26:38.351 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.364+0000 7f8c1a496640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:38.351 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.364+0000 7f8c1a496640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:38.351 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.364+0000 7f8c1a496640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:26:38.351 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.364+0000 7f8c1a496640 1 -- 172.21.15.114:0/3637590515 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8bf4005330 con 0x7f8c14071de0 2024-03-29T11:26:38.353 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T11:26:38.354 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"b0de2b9b-c45a-44c7-b6ee-e83b39a1e108","created":"2024-03-29T11:20:49.420660+0000","modified":"2024-03-29T11:20:51.700167+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:51.700167+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a64785d3-6b11-499a-b706-cc827ab96a74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5882dadb-e7a8-4604-ba9a-418013c106c8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d7e8c409-bc66-4e4a-bbdb-420bad31388e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0682989-3595-4cd6-90a5-ab9111c278e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"67068120-e3ff-4817-bd06-44feb7479a35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09e8b-8a85-4997-bdea-75d754e6ee67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2f0d9ead-f47f-4cb3-92a4-0a5ad23cd831","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fdb6a25f-6bd8-4c65-9b9b-b460ba7673a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:26:38.354 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.367+0000 7f8c10ff9640 1 -- 172.21.15.114:0/3637590515 <== mon.2 v1:172.21.15.114:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2858224613) 0x7f8c0c052640 con 0x7f8c14071de0 2024-03-29T11:26:38.354 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.367+0000 7f8c10ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:38.354 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.367+0000 7f8c10ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:26:38.357 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.370+0000 7f8c1a496640 10 librados: watch_flush enter 2024-03-29T11:26:38.357 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.370+0000 7f8c1a496640 10 librados: watch_flush exit 2024-03-29T11:26:38.357 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.370+0000 7f8c1a496640 20 client.4445.objecter shutdown clearing up homeless session... 2024-03-29T11:26:38.358 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.370+0000 7f8c1a496640 10 client.4445.objecter successfully canceled tick 2024-03-29T11:26:38.360 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.373+0000 7f8c1a496640 1 -- 172.21.15.114:0/3637590515 >> v1:172.21.15.114:6816/4038755055 conn(0x7f8c0006db80 legacy=0x7f8c00070040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:38.360 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.373+0000 7f8c1a496640 10 monclient: shutdown 2024-03-29T11:26:38.360 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.373+0000 7f8c1a496640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:38.360 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.373+0000 7f8c1a496640 1 -- 172.21.15.114:0/3637590515 >> v1:172.21.15.114:6790/0 conn(0x7f8c14071de0 legacy=0x7f8c14131f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:38.363 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.376+0000 7f8c1a496640 1 -- 172.21.15.114:0/3637590515 shutdown_connections 2024-03-29T11:26:38.363 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.376+0000 7f8c1a496640 1 -- 172.21.15.114:0/3637590515 >> 172.21.15.114:0/3637590515 conn(0x7f8c1406b660 msgr2=0x7f8c14138520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:38.363 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.376+0000 7f8c1a496640 1 -- 172.21.15.114:0/3637590515 shutdown_connections 2024-03-29T11:26:38.363 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.376+0000 7f8c1a496640 1 -- 172.21.15.114:0/3637590515 wait complete. 2024-03-29T11:26:38.364 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:38.376+0000 7f8c1a496640 1 librados: shutdown 2024-03-29T11:26:38.380 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:26:45.188 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:26:45.337 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.350+0000 7f33ad8c3640 10 monclient: get_monmap_and_config 2024-03-29T11:26:45.338 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.350+0000 7f33ad8c3640 10 monclient: build_initial_monmap 2024-03-29T11:26:45.338 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.350+0000 7f33ad8c3640 10 monclient: monmap: 2024-03-29T11:26:45.338 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:26:45.338 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:26:45.338 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:26:45.351550+0000 2024-03-29T11:26:45.338 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:26:45.351550+0000 2024-03-29T11:26:45.338 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:45.338 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:26:45.338 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:26:45.338 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T11:26:45.338 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T11:26:45.339 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:26:45.339 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.351+0000 7f33ad8c3640 1 Processor -- start 2024-03-29T11:26:45.339 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.351+0000 7f33ad8c3640 1 -- start start 2024-03-29T11:26:45.339 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.351+0000 7f33ad8c3640 10 monclient: init 2024-03-29T11:26:45.339 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.352+0000 7f33ad8c3640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:45.339 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.352+0000 7f33ad8c3640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:26:45.339 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.352+0000 7f33ad8c3640 10 monclient(hunting): picked mon.noname-b con 0x7f33a80a9ac0 addr v1:172.21.15.114:6790/0 2024-03-29T11:26:45.339 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.352+0000 7f33ad8c3640 10 monclient(hunting): picked mon.noname-a con 0x7f33a813ee90 addr v1:172.21.15.114:6789/0 2024-03-29T11:26:45.339 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.352+0000 7f33ad8c3640 10 monclient(hunting): picked mon.noname-c con 0x7f33a80af900 addr v1:172.21.15.176:6789/0 2024-03-29T11:26:45.340 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.352+0000 7f33ad8c3640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33a80afd10 con 0x7f33a813ee90 2024-03-29T11:26:45.340 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.352+0000 7f33ad8c3640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33a80af6f0 con 0x7f33a80af900 2024-03-29T11:26:45.340 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.352+0000 7f33ad8c3640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33a80aac80 con 0x7f33a80a9ac0 2024-03-29T11:26:45.340 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.352+0000 7f33ad8c3640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:45.340 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.352+0000 7f33ad8c3640 10 monclient(hunting): authenticate will time out at 1275.534439s 2024-03-29T11:26:45.342 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.352+0000 7f33a6ffd640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f33a80a9ac0 0x7f33a80adfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:51920/0 (socket says 172.21.15.114:51920) 2024-03-29T11:26:45.342 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.354+0000 7f33a6ffd640 1 -- 172.21.15.114:0/996756610 learned_addr learned my addr 172.21.15.114:0/996756610 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:26:45.342 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.354+0000 7f33a5ffb640 1 -- 172.21.15.114:0/996756610 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 384070387 0 0) 0x7f33a80aac80 con 0x7f33a80a9ac0 2024-03-29T11:26:45.342 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.354+0000 7f33a5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:26:45.342 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.354+0000 7f33a5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:45.342 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.354+0000 7f33a5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:45.342 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.355+0000 7f33a5ffb640 10 monclient(hunting): my global_id is 4448 2024-03-29T11:26:45.342 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.355+0000 7f33a5ffb640 1 -- 172.21.15.114:0/996756610 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3390003520 con 0x7f33a80a9ac0 2024-03-29T11:26:45.342 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.355+0000 7f33a5ffb640 1 -- 172.21.15.114:0/996756610 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1982331971 0 0) 0x7f3390003520 con 0x7f33a80a9ac0 2024-03-29T11:26:45.342 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.355+0000 7f33a5ffb640 1 -- 172.21.15.114:0/996756610 >> v1:172.21.15.176:6789/0 conn(0x7f33a80af900 legacy=0x7f33a8149b00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:26:45.342 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.355+0000 7f33a5ffb640 1 -- 172.21.15.114:0/996756610 >> v1:172.21.15.114:6789/0 conn(0x7f33a813ee90 legacy=0x7f33a813b5f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:26:45.345 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.358+0000 7f33a5ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:26:45.345 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.358+0000 7f33a5ffb640 1 monclient: found mon.noname-b 2024-03-29T11:26:45.345 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.358+0000 7f33a5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:45.346 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.358+0000 7f33a5ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.114:6790/0 2024-03-29T11:26:45.346 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.358+0000 7f33a5ffb640 1 -- 172.21.15.114:0/996756610 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33a80aaf60 con 0x7f33a80a9ac0 2024-03-29T11:26:45.346 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.358+0000 7f33a5ffb640 10 monclient: _finish_auth 0 2024-03-29T11:26:45.346 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.358+0000 7f33a5ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:26:45.346 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.358+0000 7f33a5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:45.346 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.358+0000 7f33a5ffb640 1 -- 172.21.15.114:0/996756610 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f339c001690 con 0x7f33a80a9ac0 2024-03-29T11:26:45.346 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.358+0000 7f33a5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:45.346 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.358+0000 7f33a5ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:26:45.346 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.358+0000 7f33a5ffb640 10 monclient: dump: 2024-03-29T11:26:45.346 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:26:45.346 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:26:45.346 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:26:45.347 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:26:45.347 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:45.347 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:26:45.347 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:26:45.347 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:26:45.347 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:26:45.347 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:26:45.347 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.358+0000 7f33ad8c3640 5 monclient: authenticate success, global_id 4448 2024-03-29T11:26:45.347 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.358+0000 7f33ad8c3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:45.348 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.359+0000 7f33a5ffb640 1 -- 172.21.15.114:0/996756610 <== mon.2 v1:172.21.15.114:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f339c001920 con 0x7f33a80a9ac0 2024-03-29T11:26:45.348 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.359+0000 7f33a5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:45.348 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.359+0000 7f33ad8c3640 10 monclient: get_monmap_and_config success 2024-03-29T11:26:45.348 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.359+0000 7f33ad8c3640 4 set_mon_vals no callback set 2024-03-29T11:26:45.348 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.360+0000 7f33a5ffb640 1 -- 172.21.15.114:0/996756610 <== mon.2 v1:172.21.15.114:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f339c001cb0 con 0x7f33a80a9ac0 2024-03-29T11:26:45.348 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.360+0000 7f33a5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:45.348 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.360+0000 7f33a5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:45.348 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.360+0000 7f33ad8c3640 10 monclient: shutdown 2024-03-29T11:26:45.349 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.360+0000 7f33a5ffb640 10 monclient: dump: 2024-03-29T11:26:45.349 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:26:45.349 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:26:45.349 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:26:45.349 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:26:45.349 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:45.349 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:26:45.349 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:26:45.349 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:26:45.349 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:26:45.349 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:26:45.349 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.360+0000 7f33ad8c3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:45.349 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.360+0000 7f33ad8c3640 1 -- 172.21.15.114:0/996756610 >> v1:172.21.15.114:6790/0 conn(0x7f33a80a9ac0 legacy=0x7f33a80adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:45.350 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.362+0000 7f33ad8c3640 1 -- 172.21.15.114:0/996756610 shutdown_connections 2024-03-29T11:26:45.350 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.362+0000 7f33ad8c3640 1 -- 172.21.15.114:0/996756610 >> 172.21.15.114:0/996756610 conn(0x7f33a80b35e0 msgr2=0x7f33a80b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:45.350 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.362+0000 7f33ad8c3640 1 -- 172.21.15.114:0/996756610 shutdown_connections 2024-03-29T11:26:45.350 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.362+0000 7f33ad8c3640 1 -- 172.21.15.114:0/996756610 wait complete. 2024-03-29T11:26:45.351 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.363+0000 7f33ad8c3640 10 monclient: build_initial_monmap 2024-03-29T11:26:45.352 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.363+0000 7f33ad8c3640 10 monclient: monmap: 2024-03-29T11:26:45.352 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:26:45.352 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:26:45.352 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T11:26:45.352 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T11:26:45.352 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:45.353 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:26:45.353 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:26:45.353 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T11:26:45.353 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T11:26:45.353 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:26:45.353 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.364+0000 7f33ad8c3640 1 librados: starting msgr at 2024-03-29T11:26:45.353 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.364+0000 7f33ad8c3640 1 librados: starting objecter 2024-03-29T11:26:45.353 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.364+0000 7f33ad8c3640 1 Processor -- start 2024-03-29T11:26:45.356 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.366+0000 7f33ad8c3640 1 -- start start 2024-03-29T11:26:45.356 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.366+0000 7f33ad8c3640 1 librados: setting wanted keys 2024-03-29T11:26:45.356 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.366+0000 7f33ad8c3640 1 librados: calling monclient init 2024-03-29T11:26:45.356 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.366+0000 7f33ad8c3640 10 monclient: init 2024-03-29T11:26:45.356 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.367+0000 7f33ad8c3640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:45.356 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.367+0000 7f33ad8c3640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:26:45.356 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.367+0000 7f33ad8c3640 10 monclient(hunting): picked mon.noname-b con 0x7f33a80a9ac0 addr v1:172.21.15.176:6789/0 2024-03-29T11:26:45.356 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.367+0000 7f33ad8c3640 10 monclient(hunting): picked mon.noname-c con 0x7f33a80af900 addr v1:172.21.15.114:6790/0 2024-03-29T11:26:45.356 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.367+0000 7f33ad8c3640 10 monclient(hunting): picked mon.noname-a con 0x7f33a813ee90 addr v1:172.21.15.114:6789/0 2024-03-29T11:26:45.356 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.367+0000 7f33ad8c3640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33a81422f0 con 0x7f33a813ee90 2024-03-29T11:26:45.357 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.367+0000 7f33ad8c3640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33a813dd80 con 0x7f33a80a9ac0 2024-03-29T11:26:45.357 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.367+0000 7f33ad8c3640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33a813e060 con 0x7f33a80af900 2024-03-29T11:26:45.357 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.367+0000 7f33ad8c3640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:45.357 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.367+0000 7f33ad8c3640 10 monclient(hunting): authenticate will time out at 1275.549272s 2024-03-29T11:26:45.357 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.367+0000 7f33a67fc640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f33a80af900 0x7f33a813fb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:33164/0 (socket says 172.21.15.114:33164) 2024-03-29T11:26:45.357 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.367+0000 7f33a67fc640 1 -- 172.21.15.114:0/4185986217 learned_addr learned my addr 172.21.15.114:0/4185986217 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:26:45.358 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.370+0000 7f338ffff640 10 client.?.objecter ms_handle_connect 0x7f33a80af900 2024-03-29T11:26:45.358 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.370+0000 7f338ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:45.358 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.370+0000 7f338ffff640 1 -- 172.21.15.114:0/4185986217 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2088657685 0 0) 0x7f33a813e060 con 0x7f33a80af900 2024-03-29T11:26:45.358 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.370+0000 7f338ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:26:45.358 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.370+0000 7f338ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:45.358 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.370+0000 7f338ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:45.359 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.370+0000 7f338ffff640 10 monclient(hunting): my global_id is 4451 2024-03-29T11:26:45.359 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.370+0000 7f338ffff640 1 -- 172.21.15.114:0/4185986217 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3394004eb0 con 0x7f33a80af900 2024-03-29T11:26:45.359 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.370+0000 7f338ffff640 1 -- 172.21.15.114:0/4185986217 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1596091569 0 0) 0x7f3394004eb0 con 0x7f33a80af900 2024-03-29T11:26:45.359 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.370+0000 7f338ffff640 1 -- 172.21.15.114:0/4185986217 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f33a813e060 con 0x7f33a80af900 2024-03-29T11:26:45.359 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.370+0000 7f338ffff640 1 -- 172.21.15.114:0/4185986217 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f33a0001ce0 con 0x7f33a80af900 2024-03-29T11:26:45.359 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.370+0000 7f338ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:45.359 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.370+0000 7f338ffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:26:45.359 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.370+0000 7f338ffff640 10 monclient(hunting): dump: 2024-03-29T11:26:45.359 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:26:45.359 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:26:45.359 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:26:45.359 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:26:45.359 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:45.359 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:26:45.360 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:26:45.360 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:26:45.360 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:26:45.360 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:26:45.360 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.370+0000 7f338ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:26:45.360 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.370+0000 7f338ffff640 1 -- 172.21.15.114:0/4185986217 <== mon.2 v1:172.21.15.114:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3736457659 0 0) 0x7f33a813e060 con 0x7f33a80af900 2024-03-29T11:26:45.360 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.370+0000 7f338ffff640 1 -- 172.21.15.114:0/4185986217 >> v1:172.21.15.176:6789/0 conn(0x7f33a80a9ac0 legacy=0x7f33a81365e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:45.360 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.370+0000 7f338ffff640 1 -- 172.21.15.114:0/4185986217 >> v1:172.21.15.114:6789/0 conn(0x7f33a813ee90 legacy=0x7f33a81e6990 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:26:45.363 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.373+0000 7f338ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:26:45.363 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.373+0000 7f338ffff640 1 monclient: found mon.c 2024-03-29T11:26:45.363 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.373+0000 7f338ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:45.363 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.373+0000 7f338ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:26:45.363 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.373+0000 7f338ffff640 1 -- 172.21.15.114:0/4185986217 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33a8140250 con 0x7f33a80af900 2024-03-29T11:26:45.363 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.373+0000 7f338ffff640 10 monclient: _finish_auth 0 2024-03-29T11:26:45.363 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.373+0000 7f338ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:26:45.363 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.373+0000 7f338ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:45.363 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.373+0000 7f33ad8c3640 5 monclient: authenticate success, global_id 4451 2024-03-29T11:26:45.363 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.373+0000 7f33ad8c3640 10 monclient: _renew_subs 2024-03-29T11:26:45.363 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.373+0000 7f33ad8c3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:26:45.364 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.373+0000 7f33ad8c3640 1 -- 172.21.15.114:0/4185986217 --> v1:172.21.15.114:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33a8135e40 con 0x7f33a80af900 2024-03-29T11:26:45.364 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.374+0000 7f33ad8c3640 10 client.4451.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:26:45.364 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.374+0000 7f33ad8c3640 10 monclient: _renew_subs 2024-03-29T11:26:45.364 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.374+0000 7f33ad8c3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:26:45.364 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.374+0000 7f33ad8c3640 1 -- 172.21.15.114:0/4185986217 --> v1:172.21.15.114:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f33a813c2a0 con 0x7f33a80af900 2024-03-29T11:26:45.364 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.374+0000 7f33ad8c3640 1 librados: init done 2024-03-29T11:26:45.364 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.374+0000 7f338ffff640 1 -- 172.21.15.114:0/4185986217 <== mon.2 v1:172.21.15.114:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f33a0001e40 con 0x7f33a80af900 2024-03-29T11:26:45.364 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.374+0000 7f338ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:45.364 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.374+0000 7f338ffff640 1 -- 172.21.15.114:0/4185986217 <== mon.2 v1:172.21.15.114:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f33a0003dc0 con 0x7f33a80af900 2024-03-29T11:26:45.364 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.374+0000 7f338ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:45.364 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.374+0000 7f338ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:45.364 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.374+0000 7f338ffff640 10 monclient: dump: 2024-03-29T11:26:45.365 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:26:45.365 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:26:45.365 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:26:45.365 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:26:45.365 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:45.365 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:26:45.365 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:26:45.365 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:26:45.365 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:26:45.365 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:26:45.365 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.374+0000 7f33a4ff9640 4 set_mon_vals no callback set 2024-03-29T11:26:45.365 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.375+0000 7f33ad8c3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:45.365 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.375+0000 7f33ad8c3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:45.365 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.375+0000 7f33ad8c3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:26:45.366 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.375+0000 7f33ad8c3640 1 -- 172.21.15.114:0/4185986217 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3388005080 con 0x7f33a80af900 2024-03-29T11:26:45.366 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.376+0000 7f338ffff640 1 -- 172.21.15.114:0/4185986217 <== mon.2 v1:172.21.15.114:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1594292461 0 0) 0x7f33a0003f20 con 0x7f33a80af900 2024-03-29T11:26:45.367 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.377+0000 7f33a6ffd640 10 monclient: get_auth_request con 0x7f339406dd00 auth_method 0 2024-03-29T11:26:45.367 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.377+0000 7f338ffff640 1 -- 172.21.15.114:0/4185986217 <== mon.2 v1:172.21.15.114:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3238987313 0 0) 0x7f33a0084e70 con 0x7f33a80af900 2024-03-29T11:26:45.367 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.377+0000 7f338ffff640 10 client.4451.objecter ms_dispatch 0x7f33a8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:26:45.367 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.377+0000 7f338ffff640 3 client.4451.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:26:45.367 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.377+0000 7f338ffff640 3 client.4451.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:26:45.367 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.377+0000 7f338ffff640 20 client.4451.objecter dump_active .. 0 homeless 2024-03-29T11:26:45.367 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.378+0000 7f338ffff640 10 client.4451.objecter ms_handle_connect 0x7f339406dd00 2024-03-29T11:26:45.374 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.387+0000 7f338ffff640 1 -- 172.21.15.114:0/4185986217 <== mon.2 v1:172.21.15.114:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f33a0050750 con 0x7f33a80af900 2024-03-29T11:26:45.374 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.387+0000 7f338ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:45.374 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.387+0000 7f338ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:26:45.699 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.711+0000 7f33ad8c3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:45.699 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.712+0000 7f33ad8c3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:45.699 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.712+0000 7f33ad8c3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:26:45.699 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.712+0000 7f33ad8c3640 1 -- 172.21.15.114:0/4185986217 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3388005330 con 0x7f33a80af900 2024-03-29T11:26:45.702 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.716+0000 7f338ffff640 1 -- 172.21.15.114:0/4185986217 <== mon.2 v1:172.21.15.114:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2858224613) 0x7f33a0052680 con 0x7f33a80af900 2024-03-29T11:26:45.702 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.716+0000 7f338ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:45.702 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.716+0000 7f338ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:26:45.705 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T11:26:45.705 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"b0de2b9b-c45a-44c7-b6ee-e83b39a1e108","created":"2024-03-29T11:20:49.420660+0000","modified":"2024-03-29T11:20:51.700167+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:51.700167+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a64785d3-6b11-499a-b706-cc827ab96a74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5882dadb-e7a8-4604-ba9a-418013c106c8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d7e8c409-bc66-4e4a-bbdb-420bad31388e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0682989-3595-4cd6-90a5-ab9111c278e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"67068120-e3ff-4817-bd06-44feb7479a35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09e8b-8a85-4997-bdea-75d754e6ee67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2f0d9ead-f47f-4cb3-92a4-0a5ad23cd831","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fdb6a25f-6bd8-4c65-9b9b-b460ba7673a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:26:45.707 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.719+0000 7f33ad8c3640 10 librados: watch_flush enter 2024-03-29T11:26:45.707 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.719+0000 7f33ad8c3640 10 librados: watch_flush exit 2024-03-29T11:26:45.707 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.719+0000 7f33ad8c3640 20 client.4451.objecter shutdown clearing up homeless session... 2024-03-29T11:26:45.707 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.719+0000 7f33ad8c3640 10 client.4451.objecter successfully canceled tick 2024-03-29T11:26:45.707 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.719+0000 7f33ad8c3640 1 -- 172.21.15.114:0/4185986217 >> v1:172.21.15.114:6816/4038755055 conn(0x7f339406dd00 legacy=0x7f33940701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:45.707 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.720+0000 7f33ad8c3640 10 monclient: shutdown 2024-03-29T11:26:45.707 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.720+0000 7f33ad8c3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:45.707 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.720+0000 7f33ad8c3640 1 -- 172.21.15.114:0/4185986217 >> v1:172.21.15.114:6790/0 conn(0x7f33a80af900 legacy=0x7f33a813fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:45.707 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.720+0000 7f33ad8c3640 1 -- 172.21.15.114:0/4185986217 shutdown_connections 2024-03-29T11:26:45.707 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.720+0000 7f33ad8c3640 1 -- 172.21.15.114:0/4185986217 >> 172.21.15.114:0/4185986217 conn(0x7f33a80b35e0 msgr2=0x7f33a80b2c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:45.710 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.723+0000 7f33ad8c3640 1 -- 172.21.15.114:0/4185986217 shutdown_connections 2024-03-29T11:26:45.710 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.724+0000 7f33ad8c3640 1 -- 172.21.15.114:0/4185986217 wait complete. 2024-03-29T11:26:45.710 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:45.724+0000 7f33ad8c3640 1 librados: shutdown 2024-03-29T11:26:45.726 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:26:52.533 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:26:52.683 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.695+0000 7f97a3377640 10 monclient: get_monmap_and_config 2024-03-29T11:26:52.683 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.695+0000 7f97a3377640 10 monclient: build_initial_monmap 2024-03-29T11:26:52.683 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.695+0000 7f97a3377640 10 monclient: monmap: 2024-03-29T11:26:52.683 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:26:52.683 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:26:52.683 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:26:52.696920+0000 2024-03-29T11:26:52.683 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:26:52.696920+0000 2024-03-29T11:26:52.683 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:52.683 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:26:52.683 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:26:52.684 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T11:26:52.684 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T11:26:52.684 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:26:52.684 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.696+0000 7f97a3377640 1 Processor -- start 2024-03-29T11:26:52.684 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.697+0000 7f97a3377640 1 -- start start 2024-03-29T11:26:52.684 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.697+0000 7f97a3377640 10 monclient: init 2024-03-29T11:26:52.686 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.697+0000 7f97a3377640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:52.686 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.697+0000 7f97a3377640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:26:52.686 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.698+0000 7f97a3377640 10 monclient(hunting): picked mon.noname-b con 0x7f979c1422b0 addr v1:172.21.15.114:6790/0 2024-03-29T11:26:52.688 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.698+0000 7f97a3377640 10 monclient(hunting): picked mon.noname-a con 0x7f979c142690 addr v1:172.21.15.114:6789/0 2024-03-29T11:26:52.688 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.700+0000 7f97a3377640 10 monclient(hunting): picked mon.noname-c con 0x7f979c0b17d0 addr v1:172.21.15.176:6789/0 2024-03-29T11:26:52.688 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.700+0000 7f97a3377640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f979c142c80 con 0x7f979c142690 2024-03-29T11:26:52.688 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.700+0000 7f97a3377640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f979c0aae40 con 0x7f979c0b17d0 2024-03-29T11:26:52.688 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.700+0000 7f97a3377640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f979c13fbd0 con 0x7f979c1422b0 2024-03-29T11:26:52.688 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.700+0000 7f97a3377640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:52.688 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.700+0000 7f97a3377640 10 monclient(hunting): authenticate will time out at 1282.882524s 2024-03-29T11:26:52.688 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.701+0000 7f97a1113640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f979c1422b0 0x7f979c13f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:55270/0 (socket says 172.21.15.114:55270) 2024-03-29T11:26:52.688 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.701+0000 7f97a1113640 1 -- 172.21.15.114:0/2939787521 learned_addr learned my addr 172.21.15.114:0/2939787521 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:26:52.689 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.701+0000 7f978ffff640 1 -- 172.21.15.114:0/2939787521 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 554937685 0 0) 0x7f979c142c80 con 0x7f979c142690 2024-03-29T11:26:52.689 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.701+0000 7f978ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:26:52.689 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.701+0000 7f978ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:52.689 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.701+0000 7f978ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:52.689 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.701+0000 7f978ffff640 10 monclient(hunting): my global_id is 4443 2024-03-29T11:26:52.689 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.701+0000 7f978ffff640 1 -- 172.21.15.114:0/2939787521 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9788003520 con 0x7f979c142690 2024-03-29T11:26:52.689 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.702+0000 7f978ffff640 1 -- 172.21.15.114:0/2939787521 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4291944192 0 0) 0x7f979c13fbd0 con 0x7f979c1422b0 2024-03-29T11:26:52.689 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.702+0000 7f978ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:26:52.689 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.702+0000 7f978ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:52.690 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.702+0000 7f978ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:52.690 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.702+0000 7f978ffff640 10 monclient(hunting): my global_id is 4454 2024-03-29T11:26:52.690 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.702+0000 7f978ffff640 1 -- 172.21.15.114:0/2939787521 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f979c142c80 con 0x7f979c1422b0 2024-03-29T11:26:52.690 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.702+0000 7f978ffff640 1 -- 172.21.15.114:0/2939787521 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 861551828 0 0) 0x7f979c142c80 con 0x7f979c1422b0 2024-03-29T11:26:52.690 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.702+0000 7f978ffff640 1 -- 172.21.15.114:0/2939787521 >> v1:172.21.15.176:6789/0 conn(0x7f979c0b17d0 legacy=0x7f979c0b2ca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:26:52.690 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.703+0000 7f978ffff640 1 -- 172.21.15.114:0/2939787521 >> v1:172.21.15.114:6789/0 conn(0x7f979c142690 legacy=0x7f979c0b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:52.690 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.703+0000 7f978ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:26:52.691 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.703+0000 7f978ffff640 1 monclient: found mon.noname-b 2024-03-29T11:26:52.691 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.704+0000 7f978ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:52.691 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.704+0000 7f978ffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.114:6790/0 2024-03-29T11:26:52.692 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.704+0000 7f978ffff640 1 -- 172.21.15.114:0/2939787521 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f979c151940 con 0x7f979c1422b0 2024-03-29T11:26:52.692 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.704+0000 7f978ffff640 10 monclient: _finish_auth 0 2024-03-29T11:26:52.692 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.704+0000 7f978ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:26:52.692 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.704+0000 7f978ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:52.692 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.704+0000 7f97a3377640 5 monclient: authenticate success, global_id 4454 2024-03-29T11:26:52.692 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.704+0000 7f97a3377640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:52.692 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.704+0000 7f978ffff640 1 -- 172.21.15.114:0/2939787521 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f9790001670 con 0x7f979c1422b0 2024-03-29T11:26:52.692 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.704+0000 7f978ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:52.692 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.704+0000 7f978ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:26:52.692 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.704+0000 7f978ffff640 10 monclient: dump: 2024-03-29T11:26:52.692 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:26:52.692 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:26:52.692 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:26:52.692 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:26:52.693 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:52.693 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:26:52.693 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:26:52.693 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:26:52.693 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:26:52.693 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:26:52.693 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.704+0000 7f97a3377640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:52.694 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.704+0000 7f978ffff640 1 -- 172.21.15.114:0/2939787521 <== mon.2 v1:172.21.15.114:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9790001890 con 0x7f979c1422b0 2024-03-29T11:26:52.694 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.704+0000 7f978ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:52.694 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.704+0000 7f97a3377640 10 monclient: get_monmap_and_config success 2024-03-29T11:26:52.694 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.704+0000 7f97a3377640 4 set_mon_vals no callback set 2024-03-29T11:26:52.694 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.705+0000 7f97a3377640 10 monclient: shutdown 2024-03-29T11:26:52.694 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.705+0000 7f97a3377640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:52.694 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.705+0000 7f97a3377640 1 -- 172.21.15.114:0/2939787521 >> v1:172.21.15.114:6790/0 conn(0x7f979c1422b0 legacy=0x7f979c13f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:52.694 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.705+0000 7f97a3377640 1 -- 172.21.15.114:0/2939787521 shutdown_connections 2024-03-29T11:26:52.694 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.705+0000 7f97a3377640 1 -- 172.21.15.114:0/2939787521 >> 172.21.15.114:0/2939787521 conn(0x7f979c066a00 msgr2=0x7f979c066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:52.694 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.706+0000 7f97a3377640 1 -- 172.21.15.114:0/2939787521 shutdown_connections 2024-03-29T11:26:52.694 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.706+0000 7f97a3377640 1 -- 172.21.15.114:0/2939787521 wait complete. 2024-03-29T11:26:52.695 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.707+0000 7f97a3377640 10 monclient: build_initial_monmap 2024-03-29T11:26:52.695 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.707+0000 7f97a3377640 10 monclient: monmap: 2024-03-29T11:26:52.695 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:26:52.695 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:26:52.695 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T11:26:52.695 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T11:26:52.696 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:52.696 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:26:52.696 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:26:52.696 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T11:26:52.696 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T11:26:52.696 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:26:52.696 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.707+0000 7f97a3377640 1 librados: starting msgr at 2024-03-29T11:26:52.696 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.707+0000 7f97a3377640 1 librados: starting objecter 2024-03-29T11:26:52.696 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.707+0000 7f97a3377640 1 Processor -- start 2024-03-29T11:26:52.696 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.707+0000 7f97a3377640 1 -- start start 2024-03-29T11:26:52.696 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.707+0000 7f97a3377640 1 librados: setting wanted keys 2024-03-29T11:26:52.696 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.707+0000 7f97a3377640 1 librados: calling monclient init 2024-03-29T11:26:52.696 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.707+0000 7f97a3377640 10 monclient: init 2024-03-29T11:26:52.697 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.708+0000 7f97a3377640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:52.697 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.708+0000 7f97a3377640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:26:52.697 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.708+0000 7f97a3377640 10 monclient(hunting): picked mon.noname-c con 0x7f979c0b17d0 addr v1:172.21.15.114:6790/0 2024-03-29T11:26:52.697 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.708+0000 7f97a3377640 10 monclient(hunting): picked mon.noname-a con 0x7f979c1422b0 addr v1:172.21.15.114:6789/0 2024-03-29T11:26:52.697 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.708+0000 7f97a3377640 10 monclient(hunting): picked mon.noname-b con 0x7f979c142690 addr v1:172.21.15.176:6789/0 2024-03-29T11:26:52.697 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.708+0000 7f97a3377640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f979c150540 con 0x7f979c1422b0 2024-03-29T11:26:52.697 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.708+0000 7f97a3377640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f979c06ed70 con 0x7f979c142690 2024-03-29T11:26:52.697 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.708+0000 7f97a3377640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f979c12f480 con 0x7f979c0b17d0 2024-03-29T11:26:52.697 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.708+0000 7f97a3377640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:52.697 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.708+0000 7f97a3377640 10 monclient(hunting): authenticate will time out at 1282.890580s 2024-03-29T11:26:52.699 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.711+0000 7f97a1113640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f979c0b17d0 0x7f979c150980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:57954/0 (socket says 172.21.15.114:57954) 2024-03-29T11:26:52.699 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.711+0000 7f97a1113640 1 -- 172.21.15.114:0/1505183659 learned_addr learned my addr 172.21.15.114:0/1505183659 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:26:52.699 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.711+0000 7f978dffb640 10 client.?.objecter ms_handle_connect 0x7f979c0b17d0 2024-03-29T11:26:52.699 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.711+0000 7f978dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:52.699 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.712+0000 7f978dffb640 1 -- 172.21.15.114:0/1505183659 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3805486143 0 0) 0x7f979c12f480 con 0x7f979c0b17d0 2024-03-29T11:26:52.699 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.712+0000 7f978dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:26:52.699 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.712+0000 7f978dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:52.699 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.712+0000 7f978dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:52.699 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.712+0000 7f978dffb640 10 monclient(hunting): my global_id is 4457 2024-03-29T11:26:52.699 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.712+0000 7f978dffb640 1 -- 172.21.15.114:0/1505183659 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9780004eb0 con 0x7f979c0b17d0 2024-03-29T11:26:52.701 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.712+0000 7f978dffb640 10 client.?.objecter ms_handle_connect 0x7f979c142690 2024-03-29T11:26:52.701 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.712+0000 7f978dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:52.701 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.712+0000 7f978dffb640 1 -- 172.21.15.114:0/1505183659 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2447653327 0 0) 0x7f9780004eb0 con 0x7f979c0b17d0 2024-03-29T11:26:52.702 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.712+0000 7f978dffb640 1 -- 172.21.15.114:0/1505183659 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f979c12f480 con 0x7f979c0b17d0 2024-03-29T11:26:52.702 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.712+0000 7f978dffb640 1 -- 172.21.15.114:0/1505183659 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f9790001630 con 0x7f979c0b17d0 2024-03-29T11:26:52.702 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.712+0000 7f978dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:52.702 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.712+0000 7f978dffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:26:52.702 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.712+0000 7f978dffb640 10 monclient(hunting): dump: 2024-03-29T11:26:52.702 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:26:52.702 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:26:52.702 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:26:52.702 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:26:52.702 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:52.703 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:26:52.703 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:26:52.703 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:26:52.703 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:26:52.703 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:26:52.703 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.713+0000 7f978dffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:26:52.703 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.713+0000 7f978dffb640 1 -- 172.21.15.114:0/1505183659 <== mon.2 v1:172.21.15.114:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3011695449 0 0) 0x7f979c12f480 con 0x7f979c0b17d0 2024-03-29T11:26:52.703 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.713+0000 7f978dffb640 1 -- 172.21.15.114:0/1505183659 >> v1:172.21.15.176:6789/0 conn(0x7f979c142690 legacy=0x7f979c13dc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:52.703 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.713+0000 7f978dffb640 1 -- 172.21.15.114:0/1505183659 >> v1:172.21.15.114:6789/0 conn(0x7f979c1422b0 legacy=0x7f979c13d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:52.703 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.713+0000 7f978dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:26:52.705 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.717+0000 7f978dffb640 1 monclient: found mon.c 2024-03-29T11:26:52.705 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.717+0000 7f978dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:52.705 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.717+0000 7f978dffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:26:52.705 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.717+0000 7f978dffb640 1 -- 172.21.15.114:0/1505183659 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f979c12f610 con 0x7f979c0b17d0 2024-03-29T11:26:52.705 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.717+0000 7f978dffb640 10 monclient: _finish_auth 0 2024-03-29T11:26:52.705 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.717+0000 7f978dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:26:52.705 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.717+0000 7f978dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:52.706 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.717+0000 7f97a3377640 5 monclient: authenticate success, global_id 4457 2024-03-29T11:26:52.706 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.717+0000 7f97a3377640 10 monclient: _renew_subs 2024-03-29T11:26:52.706 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.717+0000 7f97a3377640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:26:52.706 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.717+0000 7f97a3377640 1 -- 172.21.15.114:0/1505183659 --> v1:172.21.15.114:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f979c14ff70 con 0x7f979c0b17d0 2024-03-29T11:26:52.706 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.717+0000 7f97a3377640 10 client.4457.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:26:52.706 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.718+0000 7f97a3377640 10 monclient: _renew_subs 2024-03-29T11:26:52.706 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.718+0000 7f97a3377640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:26:52.706 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.718+0000 7f97a3377640 1 -- 172.21.15.114:0/1505183659 --> v1:172.21.15.114:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f979c0712b0 con 0x7f979c0b17d0 2024-03-29T11:26:52.707 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.718+0000 7f97a3377640 1 librados: init done 2024-03-29T11:26:52.707 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.720+0000 7f97a3377640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:52.707 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.720+0000 7f97a3377640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:52.707 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.720+0000 7f97a3377640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:26:52.707 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.720+0000 7f97a3377640 1 -- 172.21.15.114:0/1505183659 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f979c075f10 con 0x7f979c0b17d0 2024-03-29T11:26:52.710 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.720+0000 7f978dffb640 1 -- 172.21.15.114:0/1505183659 <== mon.2 v1:172.21.15.114:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9790001250 con 0x7f979c0b17d0 2024-03-29T11:26:52.711 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.720+0000 7f978dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:52.711 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.720+0000 7f978dffb640 1 -- 172.21.15.114:0/1505183659 <== mon.2 v1:172.21.15.114:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f9790004c30 con 0x7f979c0b17d0 2024-03-29T11:26:52.711 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.720+0000 7f978dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:52.711 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.720+0000 7f978dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:52.711 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.720+0000 7f978dffb640 10 monclient: dump: 2024-03-29T11:26:52.711 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:26:52.711 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:26:52.711 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:26:52.711 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:26:52.711 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:52.711 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:26:52.711 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:26:52.711 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:26:52.712 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:26:52.712 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:26:52.712 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.720+0000 7f978dffb640 1 -- 172.21.15.114:0/1505183659 <== mon.2 v1:172.21.15.114:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1594292461 0 0) 0x7f9790004e50 con 0x7f979c0b17d0 2024-03-29T11:26:52.712 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.721+0000 7f978dffb640 1 -- 172.21.15.114:0/1505183659 <== mon.2 v1:172.21.15.114:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3238987313 0 0) 0x7f9790084dd0 con 0x7f979c0b17d0 2024-03-29T11:26:52.712 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.721+0000 7f978dffb640 10 client.4457.objecter ms_dispatch 0x7f979c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:26:52.712 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.721+0000 7f978dffb640 3 client.4457.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:26:52.712 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.721+0000 7f978dffb640 3 client.4457.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:26:52.712 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.721+0000 7f978dffb640 20 client.4457.objecter dump_active .. 0 homeless 2024-03-29T11:26:52.712 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.721+0000 7f978f7fe640 4 set_mon_vals no callback set 2024-03-29T11:26:52.712 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.722+0000 7f97a0912640 10 monclient: get_auth_request con 0x7f978006dcb0 auth_method 0 2024-03-29T11:26:52.712 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.722+0000 7f978dffb640 10 client.4457.objecter ms_handle_connect 0x7f978006dcb0 2024-03-29T11:26:52.716 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.728+0000 7f978dffb640 1 -- 172.21.15.114:0/1505183659 <== mon.2 v1:172.21.15.114:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9790050730 con 0x7f979c0b17d0 2024-03-29T11:26:52.716 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.728+0000 7f978dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:52.716 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:52.728+0000 7f978dffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:26:53.039 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:53.052+0000 7f97a3377640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:53.039 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:53.052+0000 7f97a3377640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:53.039 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:53.052+0000 7f97a3377640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:26:53.039 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:53.052+0000 7f97a3377640 1 -- 172.21.15.114:0/1505183659 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f979c0b2270 con 0x7f979c0b17d0 2024-03-29T11:26:53.041 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:53.053+0000 7f978dffb640 1 -- 172.21.15.114:0/1505183659 <== mon.2 v1:172.21.15.114:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2858224613) 0x7f9790052660 con 0x7f979c0b17d0 2024-03-29T11:26:53.041 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:53.053+0000 7f978dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:53.041 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:53.053+0000 7f978dffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:26:53.043 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T11:26:53.043 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"b0de2b9b-c45a-44c7-b6ee-e83b39a1e108","created":"2024-03-29T11:20:49.420660+0000","modified":"2024-03-29T11:20:51.700167+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:51.700167+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a64785d3-6b11-499a-b706-cc827ab96a74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5882dadb-e7a8-4604-ba9a-418013c106c8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d7e8c409-bc66-4e4a-bbdb-420bad31388e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0682989-3595-4cd6-90a5-ab9111c278e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"67068120-e3ff-4817-bd06-44feb7479a35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09e8b-8a85-4997-bdea-75d754e6ee67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2f0d9ead-f47f-4cb3-92a4-0a5ad23cd831","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fdb6a25f-6bd8-4c65-9b9b-b460ba7673a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:26:53.047 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:53.059+0000 7f97a3377640 10 librados: watch_flush enter 2024-03-29T11:26:53.047 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:53.059+0000 7f97a3377640 10 librados: watch_flush exit 2024-03-29T11:26:53.047 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:53.059+0000 7f97a3377640 20 client.4457.objecter shutdown clearing up homeless session... 2024-03-29T11:26:53.047 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:53.059+0000 7f97a3377640 10 client.4457.objecter successfully canceled tick 2024-03-29T11:26:53.047 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:53.059+0000 7f97a3377640 1 -- 172.21.15.114:0/1505183659 >> v1:172.21.15.114:6816/4038755055 conn(0x7f978006dcb0 legacy=0x7f9780070170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:53.047 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:53.059+0000 7f97a3377640 10 monclient: shutdown 2024-03-29T11:26:53.047 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:53.059+0000 7f97a3377640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:53.047 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:53.059+0000 7f97a3377640 1 -- 172.21.15.114:0/1505183659 >> v1:172.21.15.114:6790/0 conn(0x7f979c0b17d0 legacy=0x7f979c150980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:53.047 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:53.059+0000 7f97a3377640 1 -- 172.21.15.114:0/1505183659 shutdown_connections 2024-03-29T11:26:53.047 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:53.059+0000 7f97a3377640 1 -- 172.21.15.114:0/1505183659 >> 172.21.15.114:0/1505183659 conn(0x7f979c066a00 msgr2=0x7f979c13c470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:53.047 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:53.060+0000 7f97a3377640 1 -- 172.21.15.114:0/1505183659 shutdown_connections 2024-03-29T11:26:53.047 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:53.060+0000 7f97a3377640 1 -- 172.21.15.114:0/1505183659 wait complete. 2024-03-29T11:26:53.047 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:26:53.060+0000 7f97a3377640 1 librados: shutdown 2024-03-29T11:26:53.063 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:26:59.871 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:27:00.022 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.033+0000 7f75a1789640 10 monclient: get_monmap_and_config 2024-03-29T11:27:00.023 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.033+0000 7f75a1789640 10 monclient: build_initial_monmap 2024-03-29T11:27:00.023 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.033+0000 7f75a1789640 10 monclient: monmap: 2024-03-29T11:27:00.023 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:27:00.023 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:27:00.023 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:27:00.034522+0000 2024-03-29T11:27:00.023 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:27:00.034522+0000 2024-03-29T11:27:00.023 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:00.023 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:27:00.024 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:27:00.024 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T11:27:00.024 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T11:27:00.024 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:27:00.024 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.034+0000 7f75a1789640 1 Processor -- start 2024-03-29T11:27:00.025 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.034+0000 7f75a1789640 1 -- start start 2024-03-29T11:27:00.026 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.034+0000 7f75a1789640 10 monclient: init 2024-03-29T11:27:00.026 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.038+0000 7f75a1789640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:00.026 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.038+0000 7f75a1789640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:27:00.026 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.038+0000 7f75a1789640 10 monclient(hunting): picked mon.noname-a con 0x7f759c0a9930 addr v1:172.21.15.114:6789/0 2024-03-29T11:27:00.026 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.038+0000 7f75a1789640 10 monclient(hunting): picked mon.noname-b con 0x7f759c149dc0 addr v1:172.21.15.114:6790/0 2024-03-29T11:27:00.026 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.038+0000 7f75a1789640 10 monclient(hunting): picked mon.noname-c con 0x7f759c14ff40 addr v1:172.21.15.176:6789/0 2024-03-29T11:27:00.026 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.038+0000 7f75a1789640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f759c0aae60 con 0x7f759c0a9930 2024-03-29T11:27:00.026 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.038+0000 7f75a1789640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f759c0b4310 con 0x7f759c14ff40 2024-03-29T11:27:00.027 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.038+0000 7f75a1789640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f759c0afdd0 con 0x7f759c149dc0 2024-03-29T11:27:00.027 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.038+0000 7f75a1789640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:00.027 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.038+0000 7f75a1789640 10 monclient(hunting): authenticate will time out at 1290.220815s 2024-03-29T11:27:00.029 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.042+0000 7f759a7fc640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f759c149dc0 0x7f759c14a240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:60105/0 (socket says 172.21.15.114:60105) 2024-03-29T11:27:00.029 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.042+0000 7f759a7fc640 1 -- 172.21.15.114:0/725272750 learned_addr learned my addr 172.21.15.114:0/725272750 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:27:00.038 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.044+0000 7f7599ffb640 1 -- 172.21.15.114:0/725272750 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 437217791 0 0) 0x7f759c0afdd0 con 0x7f759c149dc0 2024-03-29T11:27:00.038 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.044+0000 7f7599ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:27:00.039 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.044+0000 7f7599ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:00.039 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.044+0000 7f7599ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:00.039 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.044+0000 7f7599ffb640 10 monclient(hunting): my global_id is 4460 2024-03-29T11:27:00.039 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.044+0000 7f7599ffb640 1 -- 172.21.15.114:0/725272750 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7584003520 con 0x7f759c149dc0 2024-03-29T11:27:00.039 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.044+0000 7f7599ffb640 1 -- 172.21.15.114:0/725272750 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1084658665 0 0) 0x7f759c0aae60 con 0x7f759c0a9930 2024-03-29T11:27:00.039 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.044+0000 7f7599ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:27:00.039 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.044+0000 7f7599ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:00.039 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.044+0000 7f7599ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:00.039 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.044+0000 7f7599ffb640 10 monclient(hunting): my global_id is 4446 2024-03-29T11:27:00.039 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.044+0000 7f7599ffb640 1 -- 172.21.15.114:0/725272750 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f759c0afdd0 con 0x7f759c0a9930 2024-03-29T11:27:00.039 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.045+0000 7f7599ffb640 1 -- 172.21.15.114:0/725272750 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3669396423 0 0) 0x7f759c0afdd0 con 0x7f759c0a9930 2024-03-29T11:27:00.039 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.045+0000 7f7599ffb640 1 -- 172.21.15.114:0/725272750 >> v1:172.21.15.114:6790/0 conn(0x7f759c149dc0 legacy=0x7f759c14a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:00.039 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.045+0000 7f7599ffb640 1 -- 172.21.15.114:0/725272750 >> v1:172.21.15.176:6789/0 conn(0x7f759c14ff40 legacy=0x7f759c152330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:27:00.039 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.045+0000 7f7599ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:27:00.040 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.045+0000 7f7599ffb640 1 monclient: found mon.noname-a 2024-03-29T11:27:00.040 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.045+0000 7f7599ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:00.040 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.045+0000 7f7599ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.114:6789/0 2024-03-29T11:27:00.040 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.045+0000 7f7599ffb640 1 -- 172.21.15.114:0/725272750 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f759c14a950 con 0x7f759c0a9930 2024-03-29T11:27:00.040 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.045+0000 7f7599ffb640 10 monclient: _finish_auth 0 2024-03-29T11:27:00.040 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.045+0000 7f7599ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:27:00.040 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.045+0000 7f7599ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:00.040 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.045+0000 7f75a1789640 5 monclient: authenticate success, global_id 4446 2024-03-29T11:27:00.040 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.045+0000 7f75a1789640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:00.040 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.045+0000 7f7599ffb640 1 -- 172.21.15.114:0/725272750 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f758c001670 con 0x7f759c0a9930 2024-03-29T11:27:00.040 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.045+0000 7f7599ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:00.040 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.045+0000 7f7599ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:27:00.040 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.045+0000 7f7599ffb640 10 monclient: dump: 2024-03-29T11:27:00.041 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:27:00.041 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:27:00.041 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:27:00.041 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:27:00.041 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:00.041 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:27:00.041 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:27:00.041 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:27:00.041 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:27:00.041 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:27:00.041 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.045+0000 7f75a1789640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:00.041 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.045+0000 7f7599ffb640 1 -- 172.21.15.114:0/725272750 <== mon.0 v1:172.21.15.114:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f758c0017d0 con 0x7f759c0a9930 2024-03-29T11:27:00.041 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.045+0000 7f7599ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:00.041 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.045+0000 7f75a1789640 10 monclient: get_monmap_and_config success 2024-03-29T11:27:00.041 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.045+0000 7f75a1789640 4 set_mon_vals no callback set 2024-03-29T11:27:00.042 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.046+0000 7f7599ffb640 1 -- 172.21.15.114:0/725272750 <== mon.0 v1:172.21.15.114:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f758c001c20 con 0x7f759c0a9930 2024-03-29T11:27:00.042 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.046+0000 7f7599ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:00.042 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.046+0000 7f7599ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:27:00.042 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.046+0000 7f7599ffb640 10 monclient: dump: 2024-03-29T11:27:00.042 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:27:00.042 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:27:00.042 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:27:00.042 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:27:00.042 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:00.042 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:27:00.042 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:27:00.043 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:27:00.043 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:27:00.043 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:27:00.043 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.046+0000 7f75a1789640 10 monclient: shutdown 2024-03-29T11:27:00.043 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.046+0000 7f75a1789640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:00.043 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.046+0000 7f75a1789640 1 -- 172.21.15.114:0/725272750 >> v1:172.21.15.114:6789/0 conn(0x7f759c0a9930 legacy=0x7f759c143a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:00.043 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.046+0000 7f75a1789640 1 -- 172.21.15.114:0/725272750 shutdown_connections 2024-03-29T11:27:00.043 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.046+0000 7f75a1789640 1 -- 172.21.15.114:0/725272750 >> 172.21.15.114:0/725272750 conn(0x7f759c0b35e0 msgr2=0x7f759c0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:00.043 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.046+0000 7f75a1789640 1 -- 172.21.15.114:0/725272750 shutdown_connections 2024-03-29T11:27:00.043 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.046+0000 7f75a1789640 1 -- 172.21.15.114:0/725272750 wait complete. 2024-03-29T11:27:00.043 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.047+0000 7f75a1789640 10 monclient: build_initial_monmap 2024-03-29T11:27:00.044 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.047+0000 7f75a1789640 10 monclient: monmap: 2024-03-29T11:27:00.044 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:27:00.044 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:27:00.044 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T11:27:00.044 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T11:27:00.044 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:00.044 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:27:00.044 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:27:00.044 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T11:27:00.044 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T11:27:00.044 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:27:00.044 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.048+0000 7f75a1789640 1 librados: starting msgr at 2024-03-29T11:27:00.044 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.048+0000 7f75a1789640 1 librados: starting objecter 2024-03-29T11:27:00.045 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.048+0000 7f75a1789640 1 Processor -- start 2024-03-29T11:27:00.045 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.048+0000 7f75a1789640 1 -- start start 2024-03-29T11:27:00.045 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.048+0000 7f75a1789640 1 librados: setting wanted keys 2024-03-29T11:27:00.045 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.048+0000 7f75a1789640 1 librados: calling monclient init 2024-03-29T11:27:00.045 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.048+0000 7f75a1789640 10 monclient: init 2024-03-29T11:27:00.045 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.049+0000 7f75a1789640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:00.045 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.049+0000 7f75a1789640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:27:00.045 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.049+0000 7f75a1789640 10 monclient(hunting): picked mon.noname-b con 0x7f759c0a9930 addr v1:172.21.15.176:6789/0 2024-03-29T11:27:00.045 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.049+0000 7f75a1789640 10 monclient(hunting): picked mon.noname-a con 0x7f759c149dc0 addr v1:172.21.15.114:6789/0 2024-03-29T11:27:00.045 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.049+0000 7f75a1789640 10 monclient(hunting): picked mon.noname-c con 0x7f759c14ff40 addr v1:172.21.15.114:6790/0 2024-03-29T11:27:00.045 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.049+0000 7f75a1789640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f759c071ec0 con 0x7f759c149dc0 2024-03-29T11:27:00.045 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.049+0000 7f75a1789640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f759c1472f0 con 0x7f759c0a9930 2024-03-29T11:27:00.045 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.049+0000 7f75a1789640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f759c1475d0 con 0x7f759c14ff40 2024-03-29T11:27:00.045 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.049+0000 7f75a1789640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:00.045 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.049+0000 7f75a1789640 10 monclient(hunting): authenticate will time out at 1290.231913s 2024-03-29T11:27:00.046 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.053+0000 7f759affd640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f759c0a9930 0x7f759c1447f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.114:52681/0 (socket says 172.21.15.114:52681) 2024-03-29T11:27:00.047 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.053+0000 7f759affd640 1 -- 172.21.15.114:0/2978505208 learned_addr learned my addr 172.21.15.114:0/2978505208 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:27:00.047 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.054+0000 7f75837fe640 10 client.?.objecter ms_handle_connect 0x7f759c0a9930 2024-03-29T11:27:00.047 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.054+0000 7f75837fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:00.047 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.055+0000 7f75837fe640 10 client.?.objecter ms_handle_connect 0x7f759c14ff40 2024-03-29T11:27:00.047 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.055+0000 7f75837fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:00.047 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.055+0000 7f75837fe640 1 -- 172.21.15.114:0/2978505208 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1981926827 0 0) 0x7f759c1472f0 con 0x7f759c0a9930 2024-03-29T11:27:00.047 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.055+0000 7f75837fe640 10 monclient(hunting): _negotiate 2024-03-29T11:27:00.047 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.055+0000 7f75837fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:00.047 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.055+0000 7f75837fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:00.047 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.055+0000 7f75837fe640 10 monclient(hunting): my global_id is 4423 2024-03-29T11:27:00.047 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.055+0000 7f75837fe640 1 -- 172.21.15.114:0/2978505208 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7588004eb0 con 0x7f759c0a9930 2024-03-29T11:27:00.047 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.055+0000 7f75837fe640 10 client.?.objecter ms_handle_connect 0x7f759c149dc0 2024-03-29T11:27:00.047 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.055+0000 7f75837fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:00.048 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.055+0000 7f75837fe640 1 -- 172.21.15.114:0/2978505208 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3599517175 0 0) 0x7f759c071ec0 con 0x7f759c149dc0 2024-03-29T11:27:00.048 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.055+0000 7f75837fe640 10 monclient(hunting): _negotiate 2024-03-29T11:27:00.048 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.055+0000 7f75837fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:00.048 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.055+0000 7f75837fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:00.048 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.055+0000 7f75837fe640 10 monclient(hunting): my global_id is 4449 2024-03-29T11:27:00.049 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.055+0000 7f75837fe640 1 -- 172.21.15.114:0/2978505208 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f759c1472f0 con 0x7f759c149dc0 2024-03-29T11:27:00.049 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.056+0000 7f75837fe640 1 -- 172.21.15.114:0/2978505208 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1920889513 0 0) 0x7f759c1472f0 con 0x7f759c149dc0 2024-03-29T11:27:00.049 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.056+0000 7f75837fe640 1 -- 172.21.15.114:0/2978505208 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f759c071ec0 con 0x7f759c149dc0 2024-03-29T11:27:00.049 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.056+0000 7f75837fe640 1 -- 172.21.15.114:0/2978505208 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f7590001930 con 0x7f759c149dc0 2024-03-29T11:27:00.049 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.056+0000 7f75837fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:00.049 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.056+0000 7f75837fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:27:00.049 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.056+0000 7f75837fe640 10 monclient(hunting): dump: 2024-03-29T11:27:00.049 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:27:00.049 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:27:00.049 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:27:00.049 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:27:00.049 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:00.049 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:27:00.050 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:27:00.050 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:27:00.050 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:27:00.050 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:27:00.050 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.056+0000 7f75837fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:27:00.050 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.056+0000 7f75837fe640 1 -- 172.21.15.114:0/2978505208 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1199378634 0 0) 0x7f759c1475d0 con 0x7f759c14ff40 2024-03-29T11:27:00.050 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.056+0000 7f75837fe640 10 monclient(hunting): _negotiate 2024-03-29T11:27:00.050 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.056+0000 7f75837fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:00.050 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.056+0000 7f75837fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:00.051 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.056+0000 7f75837fe640 10 monclient(hunting): my global_id is 4463 2024-03-29T11:27:00.051 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.056+0000 7f75837fe640 1 -- 172.21.15.114:0/2978505208 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f759c1472f0 con 0x7f759c14ff40 2024-03-29T11:27:00.051 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.056+0000 7f75837fe640 1 -- 172.21.15.114:0/2978505208 <== mon.0 v1:172.21.15.114:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2740832743 0 0) 0x7f759c071ec0 con 0x7f759c149dc0 2024-03-29T11:27:00.051 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.056+0000 7f75837fe640 1 -- 172.21.15.114:0/2978505208 >> v1:172.21.15.114:6790/0 conn(0x7f759c14ff40 legacy=0x7f759c070dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:00.054 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.056+0000 7f75837fe640 1 -- 172.21.15.114:0/2978505208 >> v1:172.21.15.176:6789/0 conn(0x7f759c0a9930 legacy=0x7f759c1447f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:00.054 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.056+0000 7f75837fe640 10 monclient: _finish_hunting 0 2024-03-29T11:27:00.054 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.056+0000 7f75837fe640 1 monclient: found mon.a 2024-03-29T11:27:00.054 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.056+0000 7f75837fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:00.054 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.056+0000 7f75837fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T11:27:00.055 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.056+0000 7f75837fe640 1 -- 172.21.15.114:0/2978505208 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f759c1498e0 con 0x7f759c149dc0 2024-03-29T11:27:00.055 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.056+0000 7f75837fe640 10 monclient: _finish_auth 0 2024-03-29T11:27:00.055 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.056+0000 7f75837fe640 10 monclient: _check_auth_tickets 2024-03-29T11:27:00.055 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.056+0000 7f75837fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:00.055 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.057+0000 7f75837fe640 1 -- 172.21.15.114:0/2978505208 <== mon.0 v1:172.21.15.114:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f75900047e0 con 0x7f759c149dc0 2024-03-29T11:27:00.055 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.057+0000 7f75837fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:00.055 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.057+0000 7f75837fe640 1 -- 172.21.15.114:0/2978505208 <== mon.0 v1:172.21.15.114:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f7590004af0 con 0x7f759c149dc0 2024-03-29T11:27:00.055 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.057+0000 7f75837fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:00.055 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.057+0000 7f75837fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:27:00.055 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.057+0000 7f75837fe640 10 monclient: dump: 2024-03-29T11:27:00.055 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:27:00.055 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:27:00.055 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:27:00.055 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:27:00.055 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:00.056 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:27:00.056 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:27:00.056 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:27:00.056 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:27:00.056 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:27:00.056 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.057+0000 7f75a1789640 5 monclient: authenticate success, global_id 4449 2024-03-29T11:27:00.056 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.057+0000 7f75a1789640 10 monclient: _renew_subs 2024-03-29T11:27:00.056 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.057+0000 7f75a1789640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T11:27:00.056 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.057+0000 7f75a1789640 1 -- 172.21.15.114:0/2978505208 --> v1:172.21.15.114:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f759c149b30 con 0x7f759c149dc0 2024-03-29T11:27:00.056 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.057+0000 7f75a1789640 10 client.4449.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:27:00.056 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.057+0000 7f75a1789640 10 monclient: _renew_subs 2024-03-29T11:27:00.056 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.057+0000 7f75a1789640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T11:27:00.056 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.057+0000 7f75a1789640 1 -- 172.21.15.114:0/2978505208 --> v1:172.21.15.114:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f759c148e80 con 0x7f759c149dc0 2024-03-29T11:27:00.056 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.057+0000 7f75a1789640 1 librados: init done 2024-03-29T11:27:00.056 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.057+0000 7f7598ff9640 4 set_mon_vals no callback set 2024-03-29T11:27:00.057 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.060+0000 7f75837fe640 1 -- 172.21.15.114:0/2978505208 <== mon.0 v1:172.21.15.114:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1594292461 0 0) 0x7f759001aac0 con 0x7f759c149dc0 2024-03-29T11:27:00.057 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.061+0000 7f75a1789640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:00.057 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.061+0000 7f75a1789640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:00.057 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.061+0000 7f75a1789640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T11:27:00.058 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.061+0000 7f75837fe640 1 -- 172.21.15.114:0/2978505208 <== mon.0 v1:172.21.15.114:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3238987313 0 0) 0x7f7590084930 con 0x7f759c149dc0 2024-03-29T11:27:00.058 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.061+0000 7f75837fe640 10 client.4449.objecter ms_dispatch 0x7f759c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:27:00.058 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.061+0000 7f75a1789640 1 -- 172.21.15.114:0/2978505208 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f757c005080 con 0x7f759c149dc0 2024-03-29T11:27:00.058 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.061+0000 7f75837fe640 3 client.4449.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:27:00.058 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.061+0000 7f75837fe640 3 client.4449.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:27:00.058 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.061+0000 7f75837fe640 20 client.4449.objecter dump_active .. 0 homeless 2024-03-29T11:27:00.058 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.063+0000 7f759affd640 10 monclient: get_auth_request con 0x7f758806db70 auth_method 0 2024-03-29T11:27:00.058 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.064+0000 7f75837fe640 10 client.4449.objecter ms_handle_connect 0x7f758806db70 2024-03-29T11:27:00.060 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.073+0000 7f75837fe640 1 -- 172.21.15.114:0/2978505208 <== mon.0 v1:172.21.15.114:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7590050210 con 0x7f759c149dc0 2024-03-29T11:27:00.060 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.073+0000 7f75837fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:00.060 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.073+0000 7f75837fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:27:00.384 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.396+0000 7f75a1789640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:00.384 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.396+0000 7f75a1789640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:00.384 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.396+0000 7f75a1789640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T11:27:00.384 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.396+0000 7f75a1789640 1 -- 172.21.15.114:0/2978505208 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f757c005330 con 0x7f759c149dc0 2024-03-29T11:27:00.386 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.399+0000 7f75837fe640 1 -- 172.21.15.114:0/2978505208 <== mon.0 v1:172.21.15.114:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2858224613) 0x7f7590052140 con 0x7f759c149dc0 2024-03-29T11:27:00.386 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.399+0000 7f75837fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:00.386 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.399+0000 7f75837fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:27:00.388 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T11:27:00.388 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"b0de2b9b-c45a-44c7-b6ee-e83b39a1e108","created":"2024-03-29T11:20:49.420660+0000","modified":"2024-03-29T11:20:51.700167+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:51.700167+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a64785d3-6b11-499a-b706-cc827ab96a74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5882dadb-e7a8-4604-ba9a-418013c106c8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d7e8c409-bc66-4e4a-bbdb-420bad31388e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0682989-3595-4cd6-90a5-ab9111c278e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"67068120-e3ff-4817-bd06-44feb7479a35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09e8b-8a85-4997-bdea-75d754e6ee67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2f0d9ead-f47f-4cb3-92a4-0a5ad23cd831","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fdb6a25f-6bd8-4c65-9b9b-b460ba7673a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:27:00.391 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.404+0000 7f75a1789640 10 librados: watch_flush enter 2024-03-29T11:27:00.393 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.406+0000 7f75a1789640 10 librados: watch_flush exit 2024-03-29T11:27:00.393 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.406+0000 7f75a1789640 20 client.4449.objecter shutdown clearing up homeless session... 2024-03-29T11:27:00.393 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.406+0000 7f75a1789640 10 client.4449.objecter successfully canceled tick 2024-03-29T11:27:00.393 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.406+0000 7f75a1789640 1 -- 172.21.15.114:0/2978505208 >> v1:172.21.15.114:6816/4038755055 conn(0x7f758806db70 legacy=0x7f7588070030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:00.393 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.406+0000 7f75a1789640 10 monclient: shutdown 2024-03-29T11:27:00.393 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.406+0000 7f75a1789640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:00.393 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.406+0000 7f75a1789640 1 -- 172.21.15.114:0/2978505208 >> v1:172.21.15.114:6789/0 conn(0x7f759c149dc0 legacy=0x7f759c144f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:00.393 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.406+0000 7f75a1789640 1 -- 172.21.15.114:0/2978505208 shutdown_connections 2024-03-29T11:27:00.394 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.406+0000 7f75a1789640 1 -- 172.21.15.114:0/2978505208 >> 172.21.15.114:0/2978505208 conn(0x7f759c0b35e0 msgr2=0x7f759c0b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:00.394 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.406+0000 7f75a1789640 1 -- 172.21.15.114:0/2978505208 shutdown_connections 2024-03-29T11:27:00.394 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.407+0000 7f75a1789640 1 -- 172.21.15.114:0/2978505208 wait complete. 2024-03-29T11:27:00.394 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:00.407+0000 7f75a1789640 1 librados: shutdown 2024-03-29T11:27:00.412 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:27:07.220 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:27:07.382 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.394+0000 7fb81a227640 10 monclient: get_monmap_and_config 2024-03-29T11:27:07.382 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.394+0000 7fb81a227640 10 monclient: build_initial_monmap 2024-03-29T11:27:07.382 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.394+0000 7fb81a227640 10 monclient: monmap: 2024-03-29T11:27:07.382 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:27:07.382 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:27:07.382 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:27:07.395785+0000 2024-03-29T11:27:07.382 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:27:07.395785+0000 2024-03-29T11:27:07.382 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:07.382 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:27:07.383 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:27:07.383 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T11:27:07.383 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T11:27:07.383 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:27:07.384 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.395+0000 7fb81a227640 1 Processor -- start 2024-03-29T11:27:07.384 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.395+0000 7fb81a227640 1 -- start start 2024-03-29T11:27:07.384 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.395+0000 7fb81a227640 10 monclient: init 2024-03-29T11:27:07.384 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.396+0000 7fb81a227640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:07.384 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.396+0000 7fb81a227640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:27:07.384 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.396+0000 7fb81a227640 10 monclient(hunting): picked mon.noname-a con 0x7fb8140b4330 addr v1:172.21.15.114:6789/0 2024-03-29T11:27:07.385 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.396+0000 7fb81a227640 10 monclient(hunting): picked mon.noname-b con 0x7fb8141470b0 addr v1:172.21.15.114:6790/0 2024-03-29T11:27:07.385 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.396+0000 7fb81a227640 10 monclient(hunting): picked mon.noname-c con 0x7fb8141439b0 addr v1:172.21.15.176:6789/0 2024-03-29T11:27:07.385 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.396+0000 7fb81a227640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb81406f2a0 con 0x7fb8140b4330 2024-03-29T11:27:07.385 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.396+0000 7fb81a227640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb814146b90 con 0x7fb8141439b0 2024-03-29T11:27:07.385 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.396+0000 7fb81a227640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8140b2ca0 con 0x7fb8141470b0 2024-03-29T11:27:07.385 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.396+0000 7fb81a227640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:07.385 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.396+0000 7fb81a227640 10 monclient(hunting): authenticate will time out at 1297.578826s 2024-03-29T11:27:07.385 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.396+0000 7fb813fff640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7fb8141439b0 0x7fb814151d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.114:41946/0 (socket says 172.21.15.114:41946) 2024-03-29T11:27:07.385 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.396+0000 7fb813fff640 1 -- 172.21.15.114:0/3126476232 learned_addr learned my addr 172.21.15.114:0/3126476232 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:27:07.385 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.397+0000 7fb8127fc640 1 -- 172.21.15.114:0/3126476232 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3789969283 0 0) 0x7fb814146b90 con 0x7fb8141439b0 2024-03-29T11:27:07.385 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.397+0000 7fb8127fc640 10 monclient(hunting): _negotiate 2024-03-29T11:27:07.385 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.397+0000 7fb8127fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:07.385 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.397+0000 7fb8127fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:07.385 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.397+0000 7fb8127fc640 10 monclient(hunting): my global_id is 4426 2024-03-29T11:27:07.386 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.397+0000 7fb8127fc640 1 -- 172.21.15.114:0/3126476232 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7fc003520 con 0x7fb8141439b0 2024-03-29T11:27:07.394 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.397+0000 7fb8127fc640 1 -- 172.21.15.114:0/3126476232 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3172509408 0 0) 0x7fb7fc003520 con 0x7fb8141439b0 2024-03-29T11:27:07.394 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.397+0000 7fb8127fc640 1 -- 172.21.15.114:0/3126476232 >> v1:172.21.15.114:6790/0 conn(0x7fb8141470b0 legacy=0x7fb8140b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:07.394 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.397+0000 7fb8127fc640 1 -- 172.21.15.114:0/3126476232 >> v1:172.21.15.114:6789/0 conn(0x7fb8140b4330 legacy=0x7fb8140b4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:07.394 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.398+0000 7fb8127fc640 10 monclient: _finish_hunting 0 2024-03-29T11:27:07.394 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.398+0000 7fb8127fc640 1 monclient: found mon.noname-c 2024-03-29T11:27:07.394 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.398+0000 7fb8127fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:07.394 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.398+0000 7fb8127fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.176:6789/0 2024-03-29T11:27:07.394 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.398+0000 7fb8127fc640 1 -- 172.21.15.114:0/3126476232 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb81406e0c0 con 0x7fb8141439b0 2024-03-29T11:27:07.394 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.398+0000 7fb8127fc640 10 monclient: _finish_auth 0 2024-03-29T11:27:07.395 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.398+0000 7fb8127fc640 10 monclient: _check_auth_tickets 2024-03-29T11:27:07.395 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.398+0000 7fb8127fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:07.395 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.398+0000 7fb8127fc640 1 -- 172.21.15.114:0/3126476232 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fb80c001a10 con 0x7fb8141439b0 2024-03-29T11:27:07.395 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.398+0000 7fb8127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:07.395 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.398+0000 7fb8127fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:27:07.395 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.398+0000 7fb8127fc640 10 monclient: dump: 2024-03-29T11:27:07.395 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:27:07.395 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:27:07.395 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:27:07.395 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:27:07.395 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:07.395 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:27:07.395 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:27:07.395 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:27:07.396 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:27:07.396 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:27:07.396 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.398+0000 7fb81a227640 5 monclient: authenticate success, global_id 4426 2024-03-29T11:27:07.396 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.398+0000 7fb81a227640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:07.396 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.398+0000 7fb8127fc640 1 -- 172.21.15.114:0/3126476232 <== mon.1 v1:172.21.15.176:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb80c001c30 con 0x7fb8141439b0 2024-03-29T11:27:07.396 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.398+0000 7fb8127fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:07.396 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.398+0000 7fb8127fc640 1 -- 172.21.15.114:0/3126476232 <== mon.1 v1:172.21.15.176:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fb80c001fc0 con 0x7fb8141439b0 2024-03-29T11:27:07.396 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.398+0000 7fb8127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:07.396 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.398+0000 7fb8127fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:27:07.396 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.398+0000 7fb8127fc640 10 monclient: dump: 2024-03-29T11:27:07.396 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:27:07.396 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:27:07.396 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:27:07.396 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:27:07.396 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:07.397 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:27:07.397 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:27:07.397 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:27:07.397 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:27:07.397 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:27:07.397 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.398+0000 7fb81a227640 10 monclient: get_monmap_and_config success 2024-03-29T11:27:07.397 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.398+0000 7fb81a227640 4 set_mon_vals no callback set 2024-03-29T11:27:07.397 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.399+0000 7fb81a227640 10 monclient: shutdown 2024-03-29T11:27:07.397 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.399+0000 7fb81a227640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:07.397 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.399+0000 7fb81a227640 1 -- 172.21.15.114:0/3126476232 >> v1:172.21.15.176:6789/0 conn(0x7fb8141439b0 legacy=0x7fb814151d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:07.397 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.399+0000 7fb81a227640 1 -- 172.21.15.114:0/3126476232 shutdown_connections 2024-03-29T11:27:07.397 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.399+0000 7fb81a227640 1 -- 172.21.15.114:0/3126476232 >> 172.21.15.114:0/3126476232 conn(0x7fb814066a00 msgr2=0x7fb814066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:07.397 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.399+0000 7fb81a227640 1 -- 172.21.15.114:0/3126476232 shutdown_connections 2024-03-29T11:27:07.397 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.399+0000 7fb81a227640 1 -- 172.21.15.114:0/3126476232 wait complete. 2024-03-29T11:27:07.397 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.400+0000 7fb81a227640 10 monclient: build_initial_monmap 2024-03-29T11:27:07.398 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.400+0000 7fb81a227640 10 monclient: monmap: 2024-03-29T11:27:07.398 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:27:07.398 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:27:07.398 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T11:27:07.398 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T11:27:07.398 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:07.398 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:27:07.398 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:27:07.398 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T11:27:07.398 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T11:27:07.398 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:27:07.398 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.400+0000 7fb81a227640 1 librados: starting msgr at 2024-03-29T11:27:07.398 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.400+0000 7fb81a227640 1 librados: starting objecter 2024-03-29T11:27:07.398 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.401+0000 7fb81a227640 1 Processor -- start 2024-03-29T11:27:07.398 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.401+0000 7fb81a227640 1 -- start start 2024-03-29T11:27:07.399 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.401+0000 7fb81a227640 1 librados: setting wanted keys 2024-03-29T11:27:07.399 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.401+0000 7fb81a227640 1 librados: calling monclient init 2024-03-29T11:27:07.399 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.401+0000 7fb81a227640 10 monclient: init 2024-03-29T11:27:07.399 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.402+0000 7fb81a227640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:07.399 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.402+0000 7fb81a227640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:27:07.399 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.402+0000 7fb81a227640 10 monclient(hunting): picked mon.noname-b con 0x7fb8140b4330 addr v1:172.21.15.176:6789/0 2024-03-29T11:27:07.399 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.402+0000 7fb81a227640 10 monclient(hunting): picked mon.noname-c con 0x7fb8141439b0 addr v1:172.21.15.114:6790/0 2024-03-29T11:27:07.399 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.402+0000 7fb81a227640 10 monclient(hunting): picked mon.noname-a con 0x7fb8141470b0 addr v1:172.21.15.114:6789/0 2024-03-29T11:27:07.399 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.402+0000 7fb81a227640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8141db640 con 0x7fb8141470b0 2024-03-29T11:27:07.399 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.402+0000 7fb81a227640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8141276a0 con 0x7fb8140b4330 2024-03-29T11:27:07.399 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.402+0000 7fb81a227640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8141228f0 con 0x7fb8141439b0 2024-03-29T11:27:07.399 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.402+0000 7fb81a227640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:07.399 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.402+0000 7fb81a227640 10 monclient(hunting): authenticate will time out at 1297.584268s 2024-03-29T11:27:07.399 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.402+0000 7fb8137fe640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7fb8140b4330 0x7fb814125b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.114:44306/0 (socket says 172.21.15.114:44306) 2024-03-29T11:27:07.399 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.402+0000 7fb8137fe640 1 -- 172.21.15.114:0/223902686 learned_addr learned my addr 172.21.15.114:0/223902686 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:27:07.400 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.402+0000 7fb7fbfff640 10 client.?.objecter ms_handle_connect 0x7fb8140b4330 2024-03-29T11:27:07.400 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.403+0000 7fb7fbfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:07.400 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.403+0000 7fb7fbfff640 10 client.?.objecter ms_handle_connect 0x7fb8141470b0 2024-03-29T11:27:07.400 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.403+0000 7fb7fbfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:07.400 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.403+0000 7fb7fbfff640 10 client.?.objecter ms_handle_connect 0x7fb8141439b0 2024-03-29T11:27:07.400 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.403+0000 7fb7fbfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:07.400 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.403+0000 7fb7fbfff640 1 -- 172.21.15.114:0/223902686 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3763968821 0 0) 0x7fb8141228f0 con 0x7fb8141439b0 2024-03-29T11:27:07.400 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.403+0000 7fb7fbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:27:07.400 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.403+0000 7fb7fbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:07.400 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.403+0000 7fb7fbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:07.401 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.403+0000 7fb7fbfff640 10 monclient(hunting): my global_id is 4469 2024-03-29T11:27:07.401 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.403+0000 7fb7fbfff640 1 -- 172.21.15.114:0/223902686 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb800004eb0 con 0x7fb8141439b0 2024-03-29T11:27:07.401 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.403+0000 7fb7fbfff640 1 -- 172.21.15.114:0/223902686 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2615107185 0 0) 0x7fb8141db640 con 0x7fb8141470b0 2024-03-29T11:27:07.401 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.403+0000 7fb7fbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:27:07.401 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.403+0000 7fb7fbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:07.401 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.403+0000 7fb7fbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:07.401 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.403+0000 7fb7fbfff640 10 monclient(hunting): my global_id is 4455 2024-03-29T11:27:07.401 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.403+0000 7fb7fbfff640 1 -- 172.21.15.114:0/223902686 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8141228f0 con 0x7fb8141470b0 2024-03-29T11:27:07.401 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.403+0000 7fb7fbfff640 1 -- 172.21.15.114:0/223902686 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 456629150 0 0) 0x7fb8141276a0 con 0x7fb8140b4330 2024-03-29T11:27:07.401 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.403+0000 7fb7fbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:27:07.402 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.403+0000 7fb7fbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:07.402 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.403+0000 7fb7fbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:07.402 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.403+0000 7fb7fbfff640 10 monclient(hunting): my global_id is 4429 2024-03-29T11:27:07.402 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.403+0000 7fb7fbfff640 1 -- 172.21.15.114:0/223902686 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8141db640 con 0x7fb8140b4330 2024-03-29T11:27:07.402 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.404+0000 7fb7fbfff640 1 -- 172.21.15.114:0/223902686 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2096323500 0 0) 0x7fb8141db640 con 0x7fb8140b4330 2024-03-29T11:27:07.402 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.404+0000 7fb7fbfff640 1 -- 172.21.15.114:0/223902686 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb8141276a0 con 0x7fb8140b4330 2024-03-29T11:27:07.402 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.404+0000 7fb7fbfff640 1 -- 172.21.15.114:0/223902686 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fb8040018d0 con 0x7fb8140b4330 2024-03-29T11:27:07.402 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.404+0000 7fb7fbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:07.402 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.404+0000 7fb7fbfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:27:07.402 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.404+0000 7fb7fbfff640 10 monclient(hunting): dump: 2024-03-29T11:27:07.402 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:27:07.402 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:27:07.402 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:27:07.403 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:27:07.403 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:07.403 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:27:07.403 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:27:07.403 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:27:07.403 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:27:07.403 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:27:07.403 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.404+0000 7fb7fbfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:27:07.403 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.404+0000 7fb7fbfff640 1 -- 172.21.15.114:0/223902686 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1330190907 0 0) 0x7fb8141228f0 con 0x7fb8141470b0 2024-03-29T11:27:07.403 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.404+0000 7fb7fbfff640 1 -- 172.21.15.114:0/223902686 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb8141db640 con 0x7fb8141470b0 2024-03-29T11:27:07.403 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.404+0000 7fb7fbfff640 1 -- 172.21.15.114:0/223902686 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fb80c0017b0 con 0x7fb8141470b0 2024-03-29T11:27:07.403 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.404+0000 7fb7fbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:07.403 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.404+0000 7fb7fbfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:27:07.403 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.404+0000 7fb7fbfff640 10 monclient(hunting): dump: 2024-03-29T11:27:07.403 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:27:07.404 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:27:07.404 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:27:07.404 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:27:07.404 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:07.404 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:27:07.404 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:27:07.404 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:27:07.404 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:27:07.404 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:27:07.404 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.404+0000 7fb7fbfff640 1 -- 172.21.15.114:0/223902686 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3361287849 0 0) 0x7fb800004eb0 con 0x7fb8141439b0 2024-03-29T11:27:07.404 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.404+0000 7fb7fbfff640 1 -- 172.21.15.114:0/223902686 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb8141228f0 con 0x7fb8141439b0 2024-03-29T11:27:07.404 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.404+0000 7fb7fbfff640 1 -- 172.21.15.114:0/223902686 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fb808001830 con 0x7fb8141439b0 2024-03-29T11:27:07.404 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.404+0000 7fb7fbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:07.404 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.404+0000 7fb7fbfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:07.404 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.404+0000 7fb7fbfff640 10 monclient(hunting): dump: 2024-03-29T11:27:07.404 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:27:07.405 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:27:07.405 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:27:07.405 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:27:07.405 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:07.405 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:27:07.405 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:27:07.405 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:27:07.405 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:27:07.405 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:27:07.405 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.404+0000 7fb7fbfff640 1 -- 172.21.15.114:0/223902686 <== mon.1 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1695953821 0 0) 0x7fb8141276a0 con 0x7fb8140b4330 2024-03-29T11:27:07.405 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.404+0000 7fb7fbfff640 1 -- 172.21.15.114:0/223902686 >> v1:172.21.15.114:6790/0 conn(0x7fb8141439b0 legacy=0x7fb8141dbad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:07.405 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.404+0000 7fb7fbfff640 1 -- 172.21.15.114:0/223902686 >> v1:172.21.15.114:6789/0 conn(0x7fb8141470b0 legacy=0x7fb814141380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:07.405 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.404+0000 7fb7fbfff640 10 monclient: _finish_hunting 0 2024-03-29T11:27:07.405 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.404+0000 7fb7fbfff640 1 monclient: found mon.b 2024-03-29T11:27:07.406 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.404+0000 7fb7fbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:07.406 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.404+0000 7fb7fbfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:27:07.406 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.404+0000 7fb7fbfff640 1 -- 172.21.15.114:0/223902686 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb814126710 con 0x7fb8140b4330 2024-03-29T11:27:07.406 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.404+0000 7fb7fbfff640 10 monclient: _finish_auth 0 2024-03-29T11:27:07.406 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.404+0000 7fb7fbfff640 10 monclient: _check_auth_tickets 2024-03-29T11:27:07.406 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.404+0000 7fb7fbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:07.406 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.404+0000 7fb81a227640 5 monclient: authenticate success, global_id 4429 2024-03-29T11:27:07.406 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.404+0000 7fb81a227640 10 monclient: _renew_subs 2024-03-29T11:27:07.406 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.404+0000 7fb81a227640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:27:07.406 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.404+0000 7fb81a227640 1 -- 172.21.15.114:0/223902686 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb814126960 con 0x7fb8140b4330 2024-03-29T11:27:07.406 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.404+0000 7fb81a227640 10 client.4429.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:27:07.406 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.404+0000 7fb81a227640 10 monclient: _renew_subs 2024-03-29T11:27:07.406 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.404+0000 7fb81a227640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:27:07.406 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.404+0000 7fb81a227640 1 -- 172.21.15.114:0/223902686 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb814126480 con 0x7fb8140b4330 2024-03-29T11:27:07.406 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.404+0000 7fb81a227640 1 librados: init done 2024-03-29T11:27:07.407 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.405+0000 7fb7fbfff640 1 -- 172.21.15.114:0/223902686 <== mon.1 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb804001a30 con 0x7fb8140b4330 2024-03-29T11:27:07.407 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.405+0000 7fb7fbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:07.407 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.405+0000 7fb8117fa640 4 set_mon_vals no callback set 2024-03-29T11:27:07.407 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.405+0000 7fb7fbfff640 1 -- 172.21.15.114:0/223902686 <== mon.1 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fb8040037b0 con 0x7fb8140b4330 2024-03-29T11:27:07.407 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.405+0000 7fb7fbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:07.407 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.405+0000 7fb7fbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:27:07.407 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.405+0000 7fb7fbfff640 10 monclient: dump: 2024-03-29T11:27:07.407 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:27:07.407 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:27:07.407 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:27:07.407 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:27:07.408 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:07.408 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:27:07.408 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:27:07.408 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:27:07.408 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:27:07.408 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:27:07.410 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.408+0000 7fb7fbfff640 1 -- 172.21.15.114:0/223902686 <== mon.1 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1594292461 0 0) 0x7fb804003910 con 0x7fb8140b4330 2024-03-29T11:27:07.410 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.408+0000 7fb81a227640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:07.410 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.408+0000 7fb81a227640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:07.410 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.408+0000 7fb81a227640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:27:07.410 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.408+0000 7fb81a227640 1 -- 172.21.15.114:0/223902686 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7f4005080 con 0x7fb8140b4330 2024-03-29T11:27:07.410 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.409+0000 7fb812ffd640 10 monclient: get_auth_request con 0x7fb80006e240 auth_method 0 2024-03-29T11:27:07.410 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.409+0000 7fb7fbfff640 1 -- 172.21.15.114:0/223902686 <== mon.1 v1:172.21.15.176:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3238987313 0 0) 0x7fb804084980 con 0x7fb8140b4330 2024-03-29T11:27:07.410 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.409+0000 7fb7fbfff640 10 client.4429.objecter ms_dispatch 0x7fb814000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:27:07.410 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.409+0000 7fb7fbfff640 3 client.4429.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:27:07.411 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.409+0000 7fb7fbfff640 3 client.4429.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:27:07.411 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.410+0000 7fb7fbfff640 20 client.4429.objecter dump_active .. 0 homeless 2024-03-29T11:27:07.411 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.410+0000 7fb7fbfff640 10 client.4429.objecter ms_handle_connect 0x7fb80006e240 2024-03-29T11:27:07.411 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.416+0000 7fb7fbfff640 1 -- 172.21.15.114:0/223902686 <== mon.1 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb804050260 con 0x7fb8140b4330 2024-03-29T11:27:07.411 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.416+0000 7fb7fbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:07.411 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.416+0000 7fb7fbfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:27:07.779 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.791+0000 7fb81a227640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:07.779 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.791+0000 7fb81a227640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:07.779 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.791+0000 7fb81a227640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:27:07.779 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.791+0000 7fb81a227640 1 -- 172.21.15.114:0/223902686 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb7f4005330 con 0x7fb8140b4330 2024-03-29T11:27:07.781 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.793+0000 7fb7fbfff640 1 -- 172.21.15.114:0/223902686 <== mon.1 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2858224613) 0x7fb804052190 con 0x7fb8140b4330 2024-03-29T11:27:07.781 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.793+0000 7fb7fbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:07.781 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.793+0000 7fb7fbfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:27:07.781 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T11:27:07.781 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"b0de2b9b-c45a-44c7-b6ee-e83b39a1e108","created":"2024-03-29T11:20:49.420660+0000","modified":"2024-03-29T11:20:51.700167+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:51.700167+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a64785d3-6b11-499a-b706-cc827ab96a74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5882dadb-e7a8-4604-ba9a-418013c106c8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d7e8c409-bc66-4e4a-bbdb-420bad31388e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0682989-3595-4cd6-90a5-ab9111c278e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"67068120-e3ff-4817-bd06-44feb7479a35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09e8b-8a85-4997-bdea-75d754e6ee67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2f0d9ead-f47f-4cb3-92a4-0a5ad23cd831","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fdb6a25f-6bd8-4c65-9b9b-b460ba7673a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:27:07.785 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.797+0000 7fb81a227640 10 librados: watch_flush enter 2024-03-29T11:27:07.785 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.798+0000 7fb81a227640 10 librados: watch_flush exit 2024-03-29T11:27:07.785 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.798+0000 7fb81a227640 20 client.4429.objecter shutdown clearing up homeless session... 2024-03-29T11:27:07.785 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.798+0000 7fb81a227640 10 client.4429.objecter successfully canceled tick 2024-03-29T11:27:07.788 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.800+0000 7fb81a227640 1 -- 172.21.15.114:0/223902686 >> v1:172.21.15.114:6816/4038755055 conn(0x7fb80006e240 legacy=0x7fb800070700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:07.788 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.800+0000 7fb81a227640 10 monclient: shutdown 2024-03-29T11:27:07.788 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.800+0000 7fb81a227640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:07.788 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.800+0000 7fb81a227640 1 -- 172.21.15.114:0/223902686 >> v1:172.21.15.176:6789/0 conn(0x7fb8140b4330 legacy=0x7fb814125b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:07.788 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.801+0000 7fb81a227640 1 -- 172.21.15.114:0/223902686 shutdown_connections 2024-03-29T11:27:07.788 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.801+0000 7fb81a227640 1 -- 172.21.15.114:0/223902686 >> 172.21.15.114:0/223902686 conn(0x7fb814066a00 msgr2=0x7fb8140b4710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:07.789 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.801+0000 7fb81a227640 1 -- 172.21.15.114:0/223902686 shutdown_connections 2024-03-29T11:27:07.789 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.801+0000 7fb81a227640 1 -- 172.21.15.114:0/223902686 wait complete. 2024-03-29T11:27:07.789 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:07.801+0000 7fb81a227640 1 librados: shutdown 2024-03-29T11:27:07.807 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:27:14.614 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:27:14.762 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.773+0000 7f8450bed640 10 monclient: get_monmap_and_config 2024-03-29T11:27:14.762 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.773+0000 7f8450bed640 10 monclient: build_initial_monmap 2024-03-29T11:27:14.762 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.773+0000 7f8450bed640 10 monclient: monmap: 2024-03-29T11:27:14.762 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:27:14.762 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:27:14.762 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:27:14.774309+0000 2024-03-29T11:27:14.762 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:27:14.774309+0000 2024-03-29T11:27:14.762 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:14.762 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:27:14.762 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:27:14.762 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T11:27:14.763 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T11:27:14.763 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:27:14.763 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.774+0000 7f8450bed640 1 Processor -- start 2024-03-29T11:27:14.763 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.774+0000 7f8450bed640 1 -- start start 2024-03-29T11:27:14.763 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.774+0000 7f8450bed640 10 monclient: init 2024-03-29T11:27:14.763 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.775+0000 7f8450bed640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:14.763 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.775+0000 7f8450bed640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:27:14.764 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.775+0000 7f8450bed640 10 monclient(hunting): picked mon.noname-b con 0x7f844c1457f0 addr v1:172.21.15.114:6790/0 2024-03-29T11:27:14.764 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.775+0000 7f8450bed640 10 monclient(hunting): picked mon.noname-a con 0x7f844c13f650 addr v1:172.21.15.114:6789/0 2024-03-29T11:27:14.764 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.775+0000 7f8450bed640 10 monclient(hunting): picked mon.noname-c con 0x7f844c0a9d70 addr v1:172.21.15.176:6789/0 2024-03-29T11:27:14.764 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.775+0000 7f8450bed640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f844c0aaef0 con 0x7f844c13f650 2024-03-29T11:27:14.764 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.775+0000 7f8450bed640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f844c145bd0 con 0x7f844c0a9d70 2024-03-29T11:27:14.764 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.775+0000 7f8450bed640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f844c145eb0 con 0x7f844c1457f0 2024-03-29T11:27:14.764 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.775+0000 7f8450bed640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:14.764 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.775+0000 7f8450bed640 10 monclient(hunting): authenticate will time out at 1304.957805s 2024-03-29T11:27:14.764 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.776+0000 7f8449d9b640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f844c13f650 0x7f844c0b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:37614/0 (socket says 172.21.15.114:37614) 2024-03-29T11:27:14.764 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.776+0000 7f8449d9b640 1 -- 172.21.15.114:0/579827477 learned_addr learned my addr 172.21.15.114:0/579827477 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:27:14.765 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.777+0000 7f844959a640 1 -- 172.21.15.114:0/579827477 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1118762060 0 0) 0x7f844c0aaef0 con 0x7f844c13f650 2024-03-29T11:27:14.765 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.777+0000 7f844959a640 10 monclient(hunting): _negotiate 2024-03-29T11:27:14.765 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.777+0000 7f844959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:14.765 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.778+0000 7f844959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:14.765 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.778+0000 7f844959a640 10 monclient(hunting): my global_id is 4458 2024-03-29T11:27:14.768 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.778+0000 7f844959a640 1 -- 172.21.15.114:0/579827477 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8438003520 con 0x7f844c13f650 2024-03-29T11:27:14.769 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.778+0000 7f844959a640 1 -- 172.21.15.114:0/579827477 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1633992633 0 0) 0x7f844c145eb0 con 0x7f844c1457f0 2024-03-29T11:27:14.769 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.778+0000 7f844959a640 10 monclient(hunting): _negotiate 2024-03-29T11:27:14.769 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.779+0000 7f844959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:14.769 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.779+0000 7f844959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:14.769 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.779+0000 7f844959a640 10 monclient(hunting): my global_id is 4472 2024-03-29T11:27:14.770 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.779+0000 7f844959a640 1 -- 172.21.15.114:0/579827477 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f844c0aaef0 con 0x7f844c1457f0 2024-03-29T11:27:14.770 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.779+0000 7f844959a640 1 -- 172.21.15.114:0/579827477 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1585452428 0 0) 0x7f844c145bd0 con 0x7f844c0a9d70 2024-03-29T11:27:14.770 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.779+0000 7f844959a640 10 monclient(hunting): _negotiate 2024-03-29T11:27:14.770 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.779+0000 7f844959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:14.770 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.779+0000 7f844959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:14.770 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.779+0000 7f844959a640 10 monclient(hunting): my global_id is 4432 2024-03-29T11:27:14.770 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.779+0000 7f844959a640 1 -- 172.21.15.114:0/579827477 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f844c145eb0 con 0x7f844c0a9d70 2024-03-29T11:27:14.770 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.779+0000 7f844959a640 1 -- 172.21.15.114:0/579827477 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 192040191 0 0) 0x7f8438003520 con 0x7f844c13f650 2024-03-29T11:27:14.770 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.779+0000 7f844959a640 1 -- 172.21.15.114:0/579827477 >> v1:172.21.15.114:6790/0 conn(0x7f844c1457f0 legacy=0x7f844c13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:14.770 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.780+0000 7f844959a640 1 -- 172.21.15.114:0/579827477 >> v1:172.21.15.176:6789/0 conn(0x7f844c0a9d70 legacy=0x7f844c14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:14.770 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.780+0000 7f844959a640 10 monclient: _finish_hunting 0 2024-03-29T11:27:14.770 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.780+0000 7f844959a640 1 monclient: found mon.noname-a 2024-03-29T11:27:14.771 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.780+0000 7f844959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:14.771 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.780+0000 7f844959a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.114:6789/0 2024-03-29T11:27:14.771 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.780+0000 7f844959a640 1 -- 172.21.15.114:0/579827477 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f844c146340 con 0x7f844c13f650 2024-03-29T11:27:14.771 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.780+0000 7f844959a640 10 monclient: _finish_auth 0 2024-03-29T11:27:14.771 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.780+0000 7f844959a640 10 monclient: _check_auth_tickets 2024-03-29T11:27:14.771 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.780+0000 7f844959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:14.771 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.780+0000 7f844959a640 1 -- 172.21.15.114:0/579827477 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f8434001690 con 0x7f844c13f650 2024-03-29T11:27:14.771 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.780+0000 7f844959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:14.771 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.780+0000 7f844959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:27:14.771 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.780+0000 7f844959a640 10 monclient: dump: 2024-03-29T11:27:14.771 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:27:14.771 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:27:14.771 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:27:14.771 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:27:14.772 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:14.772 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:27:14.773 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:27:14.773 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:27:14.773 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:27:14.773 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:27:14.773 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.780+0000 7f8450bed640 5 monclient: authenticate success, global_id 4458 2024-03-29T11:27:14.773 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.780+0000 7f8450bed640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:14.773 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.781+0000 7f844959a640 1 -- 172.21.15.114:0/579827477 <== mon.0 v1:172.21.15.114:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f84340017f0 con 0x7f844c13f650 2024-03-29T11:27:14.774 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.781+0000 7f844959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:14.774 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.781+0000 7f844959a640 1 -- 172.21.15.114:0/579827477 <== mon.0 v1:172.21.15.114:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f8434001c40 con 0x7f844c13f650 2024-03-29T11:27:14.774 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.781+0000 7f844959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:14.774 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.781+0000 7f844959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:27:14.774 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.781+0000 7f844959a640 10 monclient: dump: 2024-03-29T11:27:14.774 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:27:14.774 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:27:14.774 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:27:14.774 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:27:14.774 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:14.774 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:27:14.774 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:27:14.774 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:27:14.774 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:27:14.775 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:27:14.775 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.781+0000 7f8450bed640 10 monclient: get_monmap_and_config success 2024-03-29T11:27:14.775 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.781+0000 7f8450bed640 4 set_mon_vals no callback set 2024-03-29T11:27:14.781 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.782+0000 7f8450bed640 10 monclient: shutdown 2024-03-29T11:27:14.781 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.782+0000 7f8450bed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:14.782 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.782+0000 7f8450bed640 1 -- 172.21.15.114:0/579827477 >> v1:172.21.15.114:6789/0 conn(0x7f844c13f650 legacy=0x7f844c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:14.782 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.782+0000 7f8450bed640 1 -- 172.21.15.114:0/579827477 shutdown_connections 2024-03-29T11:27:14.782 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.782+0000 7f8450bed640 1 -- 172.21.15.114:0/579827477 >> 172.21.15.114:0/579827477 conn(0x7f844c066a00 msgr2=0x7f844c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:14.782 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.783+0000 7f8450bed640 1 -- 172.21.15.114:0/579827477 shutdown_connections 2024-03-29T11:27:14.782 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.783+0000 7f8450bed640 1 -- 172.21.15.114:0/579827477 wait complete. 2024-03-29T11:27:14.782 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.784+0000 7f8450bed640 10 monclient: build_initial_monmap 2024-03-29T11:27:14.782 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.784+0000 7f8450bed640 10 monclient: monmap: 2024-03-29T11:27:14.782 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:27:14.782 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:27:14.782 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T11:27:14.782 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T11:27:14.782 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:14.782 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:27:14.782 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:27:14.783 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T11:27:14.783 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T11:27:14.783 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:27:14.783 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.784+0000 7f8450bed640 1 librados: starting msgr at 2024-03-29T11:27:14.783 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.784+0000 7f8450bed640 1 librados: starting objecter 2024-03-29T11:27:14.783 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.784+0000 7f8450bed640 1 Processor -- start 2024-03-29T11:27:14.783 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.784+0000 7f8450bed640 1 -- start start 2024-03-29T11:27:14.783 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.784+0000 7f8450bed640 1 librados: setting wanted keys 2024-03-29T11:27:14.783 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.784+0000 7f8450bed640 1 librados: calling monclient init 2024-03-29T11:27:14.783 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.784+0000 7f8450bed640 10 monclient: init 2024-03-29T11:27:14.783 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.785+0000 7f8450bed640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:14.783 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.785+0000 7f8450bed640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:27:14.783 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.785+0000 7f8450bed640 10 monclient(hunting): picked mon.noname-a con 0x7f844c0a9d70 addr v1:172.21.15.114:6789/0 2024-03-29T11:27:14.783 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.785+0000 7f8450bed640 10 monclient(hunting): picked mon.noname-c con 0x7f844c13f650 addr v1:172.21.15.114:6790/0 2024-03-29T11:27:14.783 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.785+0000 7f8450bed640 10 monclient(hunting): picked mon.noname-b con 0x7f844c1457f0 addr v1:172.21.15.176:6789/0 2024-03-29T11:27:14.784 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.785+0000 7f8450bed640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f844c0ad210 con 0x7f844c0a9d70 2024-03-29T11:27:14.784 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.785+0000 7f8450bed640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f844c0ac440 con 0x7f844c1457f0 2024-03-29T11:27:14.784 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.785+0000 7f8450bed640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f844c142cd0 con 0x7f844c13f650 2024-03-29T11:27:14.784 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.785+0000 7f8450bed640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:14.784 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.785+0000 7f8450bed640 10 monclient(hunting): authenticate will time out at 1304.967816s 2024-03-29T11:27:14.784 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.785+0000 7f8449d9b640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f844c13f650 0x7f844c1d7810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:51072/0 (socket says 172.21.15.114:51072) 2024-03-29T11:27:14.784 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.785+0000 7f8449d9b640 1 -- 172.21.15.114:0/3861372338 learned_addr learned my addr 172.21.15.114:0/3861372338 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:27:14.784 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.786+0000 7f8442ffd640 10 client.?.objecter ms_handle_connect 0x7f844c13f650 2024-03-29T11:27:14.784 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.786+0000 7f8442ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:14.784 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.786+0000 7f8442ffd640 10 client.?.objecter ms_handle_connect 0x7f844c0a9d70 2024-03-29T11:27:14.784 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.786+0000 7f8442ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:14.784 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.786+0000 7f8442ffd640 1 -- 172.21.15.114:0/3861372338 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1669641020 0 0) 0x7f844c142cd0 con 0x7f844c13f650 2024-03-29T11:27:14.784 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.786+0000 7f8442ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:27:14.784 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.786+0000 7f8442ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:14.785 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.786+0000 7f8442ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:14.785 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.786+0000 7f8442ffd640 10 monclient(hunting): my global_id is 4475 2024-03-29T11:27:14.785 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.786+0000 7f8442ffd640 1 -- 172.21.15.114:0/3861372338 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f842c004eb0 con 0x7f844c13f650 2024-03-29T11:27:14.785 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.786+0000 7f8442ffd640 10 client.?.objecter ms_handle_connect 0x7f844c1457f0 2024-03-29T11:27:14.785 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.786+0000 7f8442ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:14.785 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.786+0000 7f8442ffd640 1 -- 172.21.15.114:0/3861372338 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2152005975 0 0) 0x7f844c0ad210 con 0x7f844c0a9d70 2024-03-29T11:27:14.785 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.786+0000 7f8442ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:27:14.785 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.786+0000 7f8442ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:14.785 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.786+0000 7f8442ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:14.785 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.786+0000 7f8442ffd640 10 monclient(hunting): my global_id is 4461 2024-03-29T11:27:14.786 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.786+0000 7f8442ffd640 1 -- 172.21.15.114:0/3861372338 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f844c142cd0 con 0x7f844c0a9d70 2024-03-29T11:27:14.786 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.787+0000 7f8442ffd640 1 -- 172.21.15.114:0/3861372338 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3905404261 0 0) 0x7f844c0ac440 con 0x7f844c1457f0 2024-03-29T11:27:14.786 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.787+0000 7f8442ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:27:14.786 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.787+0000 7f8442ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:14.786 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.787+0000 7f8442ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:14.786 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.787+0000 7f8442ffd640 10 monclient(hunting): my global_id is 4435 2024-03-29T11:27:14.786 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.787+0000 7f8442ffd640 1 -- 172.21.15.114:0/3861372338 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f844c0ad210 con 0x7f844c1457f0 2024-03-29T11:27:14.786 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.787+0000 7f8442ffd640 1 -- 172.21.15.114:0/3861372338 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1633941370 0 0) 0x7f842c004eb0 con 0x7f844c13f650 2024-03-29T11:27:14.786 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.787+0000 7f8442ffd640 1 -- 172.21.15.114:0/3861372338 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f844c0ac440 con 0x7f844c13f650 2024-03-29T11:27:14.786 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.787+0000 7f8442ffd640 1 -- 172.21.15.114:0/3861372338 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f8434001bc0 con 0x7f844c13f650 2024-03-29T11:27:14.786 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.787+0000 7f8442ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:14.786 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.787+0000 7f8442ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:27:14.786 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.787+0000 7f8442ffd640 10 monclient(hunting): dump: 2024-03-29T11:27:14.786 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:27:14.787 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:27:14.787 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:27:14.787 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:27:14.787 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:14.787 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:27:14.787 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:27:14.787 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:27:14.787 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:27:14.787 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:27:14.787 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.787+0000 7f8442ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:27:14.787 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.787+0000 7f8442ffd640 1 -- 172.21.15.114:0/3861372338 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 126697246 0 0) 0x7f844c142cd0 con 0x7f844c0a9d70 2024-03-29T11:27:14.788 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.787+0000 7f8442ffd640 1 -- 172.21.15.114:0/3861372338 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f842c004eb0 con 0x7f844c0a9d70 2024-03-29T11:27:14.788 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.787+0000 7f8442ffd640 1 -- 172.21.15.114:0/3861372338 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f8444001a90 con 0x7f844c0a9d70 2024-03-29T11:27:14.788 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.787+0000 7f8442ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:14.788 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.787+0000 7f8442ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:27:14.788 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.787+0000 7f8442ffd640 10 monclient(hunting): dump: 2024-03-29T11:27:14.788 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:27:14.788 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:27:14.788 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:27:14.788 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:27:14.788 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:14.788 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:27:14.788 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:27:14.788 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:27:14.788 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:27:14.789 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:27:14.789 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.788+0000 7f8442ffd640 1 -- 172.21.15.114:0/3861372338 <== mon.2 v1:172.21.15.114:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4174647626 0 0) 0x7f844c0ac440 con 0x7f844c13f650 2024-03-29T11:27:14.789 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.788+0000 7f8442ffd640 1 -- 172.21.15.114:0/3861372338 >> v1:172.21.15.176:6789/0 conn(0x7f844c1457f0 legacy=0x7f844c0b0c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:14.789 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.788+0000 7f8442ffd640 1 -- 172.21.15.114:0/3861372338 >> v1:172.21.15.114:6789/0 conn(0x7f844c0a9d70 legacy=0x7f844c0abd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:14.789 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.788+0000 7f8442ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:27:14.789 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.788+0000 7f8442ffd640 1 monclient: found mon.c 2024-03-29T11:27:14.789 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.788+0000 7f8442ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:14.789 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.788+0000 7f8442ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:27:14.789 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.788+0000 7f8442ffd640 1 -- 172.21.15.114:0/3861372338 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f844c1d72f0 con 0x7f844c13f650 2024-03-29T11:27:14.789 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.788+0000 7f8442ffd640 10 monclient: _finish_auth 0 2024-03-29T11:27:14.789 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.788+0000 7f8442ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:27:14.790 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.788+0000 7f8442ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:14.790 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.788+0000 7f8450bed640 5 monclient: authenticate success, global_id 4475 2024-03-29T11:27:14.790 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.788+0000 7f8450bed640 10 monclient: _renew_subs 2024-03-29T11:27:14.790 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.788+0000 7f8450bed640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:27:14.790 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.788+0000 7f8450bed640 1 -- 172.21.15.114:0/3861372338 --> v1:172.21.15.114:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f844c13fda0 con 0x7f844c13f650 2024-03-29T11:27:14.790 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.788+0000 7f8450bed640 10 client.4475.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:27:14.790 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.788+0000 7f8450bed640 10 monclient: _renew_subs 2024-03-29T11:27:14.790 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.788+0000 7f8450bed640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:27:14.790 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.788+0000 7f8450bed640 1 -- 172.21.15.114:0/3861372338 --> v1:172.21.15.114:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f844c1d8100 con 0x7f844c13f650 2024-03-29T11:27:14.790 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.788+0000 7f8450bed640 1 librados: init done 2024-03-29T11:27:14.790 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.788+0000 7f8442ffd640 1 -- 172.21.15.114:0/3861372338 <== mon.2 v1:172.21.15.114:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8434004bf0 con 0x7f844c13f650 2024-03-29T11:27:14.790 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.788+0000 7f8442ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:14.791 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.788+0000 7f8448d99640 4 set_mon_vals no callback set 2024-03-29T11:27:14.791 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.789+0000 7f8442ffd640 1 -- 172.21.15.114:0/3861372338 <== mon.2 v1:172.21.15.114:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f8434004f00 con 0x7f844c13f650 2024-03-29T11:27:14.791 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.789+0000 7f8442ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:14.791 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.789+0000 7f8442ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:14.791 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.789+0000 7f8442ffd640 10 monclient: dump: 2024-03-29T11:27:14.791 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:27:14.791 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:27:14.791 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:27:14.791 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:27:14.791 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:14.791 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:27:14.792 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:27:14.792 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:27:14.792 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:27:14.792 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:27:14.792 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.790+0000 7f8450bed640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:14.792 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.790+0000 7f8450bed640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:14.792 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.790+0000 7f8450bed640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:27:14.792 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.790+0000 7f8450bed640 1 -- 172.21.15.114:0/3861372338 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8430005080 con 0x7f844c13f650 2024-03-29T11:27:14.792 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.791+0000 7f8442ffd640 1 -- 172.21.15.114:0/3861372338 <== mon.2 v1:172.21.15.114:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1594292461 0 0) 0x7f843401aed0 con 0x7f844c13f650 2024-03-29T11:27:14.792 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.792+0000 7f8442ffd640 1 -- 172.21.15.114:0/3861372338 <== mon.2 v1:172.21.15.114:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3238987313 0 0) 0x7f8434084bb0 con 0x7f844c13f650 2024-03-29T11:27:14.792 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.792+0000 7f8442ffd640 10 client.4475.objecter ms_dispatch 0x7f844c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:27:14.793 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.792+0000 7f8442ffd640 3 client.4475.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:27:14.793 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.792+0000 7f8442ffd640 3 client.4475.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:27:14.793 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.792+0000 7f8442ffd640 20 client.4475.objecter dump_active .. 0 homeless 2024-03-29T11:27:14.793 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.792+0000 7f844a59c640 10 monclient: get_auth_request con 0x7f842c06df60 auth_method 0 2024-03-29T11:27:14.793 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.792+0000 7f8442ffd640 10 client.4475.objecter ms_handle_connect 0x7f842c06df60 2024-03-29T11:27:14.793 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.799+0000 7f8442ffd640 1 -- 172.21.15.114:0/3861372338 <== mon.2 v1:172.21.15.114:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8434050490 con 0x7f844c13f650 2024-03-29T11:27:14.794 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.799+0000 7f8442ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:14.794 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:14.799+0000 7f8442ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:27:15.153 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:15.166+0000 7f8450bed640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:15.154 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:15.166+0000 7f8450bed640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:15.154 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:15.166+0000 7f8450bed640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:27:15.154 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:15.166+0000 7f8450bed640 1 -- 172.21.15.114:0/3861372338 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8430005330 con 0x7f844c13f650 2024-03-29T11:27:15.154 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:15.166+0000 7f8442ffd640 1 -- 172.21.15.114:0/3861372338 <== mon.2 v1:172.21.15.114:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2858224613) 0x7f84340523c0 con 0x7f844c13f650 2024-03-29T11:27:15.154 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:15.166+0000 7f8442ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:15.154 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:15.166+0000 7f8442ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:27:15.155 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T11:27:15.155 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"b0de2b9b-c45a-44c7-b6ee-e83b39a1e108","created":"2024-03-29T11:20:49.420660+0000","modified":"2024-03-29T11:20:51.700167+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:51.700167+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a64785d3-6b11-499a-b706-cc827ab96a74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5882dadb-e7a8-4604-ba9a-418013c106c8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d7e8c409-bc66-4e4a-bbdb-420bad31388e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0682989-3595-4cd6-90a5-ab9111c278e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"67068120-e3ff-4817-bd06-44feb7479a35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09e8b-8a85-4997-bdea-75d754e6ee67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2f0d9ead-f47f-4cb3-92a4-0a5ad23cd831","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fdb6a25f-6bd8-4c65-9b9b-b460ba7673a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:27:15.158 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:15.170+0000 7f8450bed640 10 librados: watch_flush enter 2024-03-29T11:27:15.158 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:15.170+0000 7f8450bed640 10 librados: watch_flush exit 2024-03-29T11:27:15.159 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:15.170+0000 7f8450bed640 20 client.4475.objecter shutdown clearing up homeless session... 2024-03-29T11:27:15.159 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:15.170+0000 7f8450bed640 10 client.4475.objecter successfully canceled tick 2024-03-29T11:27:15.159 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:15.170+0000 7f8450bed640 1 -- 172.21.15.114:0/3861372338 >> v1:172.21.15.114:6816/4038755055 conn(0x7f842c06df60 legacy=0x7f842c070420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:15.159 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:15.170+0000 7f8450bed640 10 monclient: shutdown 2024-03-29T11:27:15.159 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:15.170+0000 7f8450bed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:15.159 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:15.170+0000 7f8450bed640 1 -- 172.21.15.114:0/3861372338 >> v1:172.21.15.114:6790/0 conn(0x7f844c13f650 legacy=0x7f844c1d7810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:15.159 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:15.170+0000 7f8450bed640 1 -- 172.21.15.114:0/3861372338 shutdown_connections 2024-03-29T11:27:15.159 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:15.170+0000 7f8450bed640 1 -- 172.21.15.114:0/3861372338 >> 172.21.15.114:0/3861372338 conn(0x7f844c066a00 msgr2=0x7f844c066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:15.159 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:15.171+0000 7f8450bed640 1 -- 172.21.15.114:0/3861372338 shutdown_connections 2024-03-29T11:27:15.159 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:15.171+0000 7f8450bed640 1 -- 172.21.15.114:0/3861372338 wait complete. 2024-03-29T11:27:15.159 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:15.171+0000 7f8450bed640 1 librados: shutdown 2024-03-29T11:27:15.175 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:27:21.983 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:27:22.135 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.145+0000 7fe79ae5f640 10 monclient: get_monmap_and_config 2024-03-29T11:27:22.135 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.145+0000 7fe79ae5f640 10 monclient: build_initial_monmap 2024-03-29T11:27:22.135 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.145+0000 7fe79ae5f640 10 monclient: monmap: 2024-03-29T11:27:22.135 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:27:22.136 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:27:22.136 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:27:22.146653+0000 2024-03-29T11:27:22.136 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:27:22.146653+0000 2024-03-29T11:27:22.136 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:22.136 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:27:22.136 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:27:22.136 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T11:27:22.136 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T11:27:22.136 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:27:22.138 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.146+0000 7fe79ae5f640 1 Processor -- start 2024-03-29T11:27:22.138 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.146+0000 7fe79ae5f640 1 -- start start 2024-03-29T11:27:22.138 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.146+0000 7fe79ae5f640 10 monclient: init 2024-03-29T11:27:22.138 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.147+0000 7fe79ae5f640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:22.138 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.147+0000 7fe79ae5f640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:27:22.138 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.147+0000 7fe79ae5f640 10 monclient(hunting): picked mon.noname-c con 0x7fe78c006890 addr v1:172.21.15.176:6789/0 2024-03-29T11:27:22.138 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.147+0000 7fe79ae5f640 10 monclient(hunting): picked mon.noname-b con 0x7fe78c006c70 addr v1:172.21.15.114:6790/0 2024-03-29T11:27:22.138 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.147+0000 7fe79ae5f640 10 monclient(hunting): picked mon.noname-a con 0x7fe78c02b4c0 addr v1:172.21.15.114:6789/0 2024-03-29T11:27:22.138 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.147+0000 7fe79ae5f640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe78c02c830 con 0x7fe78c02b4c0 2024-03-29T11:27:22.138 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.147+0000 7fe79ae5f640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe78c02afa0 con 0x7fe78c006890 2024-03-29T11:27:22.139 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.147+0000 7fe79ae5f640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe78c02aba0 con 0x7fe78c006c70 2024-03-29T11:27:22.139 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.147+0000 7fe79ae5f640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:22.139 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.147+0000 7fe79ae5f640 10 monclient(hunting): authenticate will time out at 1312.329842s 2024-03-29T11:27:22.139 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.152+0000 7fe79965c640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7fe78c006c70 0x7fe78c0c6d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:35349/0 (socket says 172.21.15.114:35349) 2024-03-29T11:27:22.140 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.152+0000 7fe79965c640 1 -- 172.21.15.114:0/2623037764 learned_addr learned my addr 172.21.15.114:0/2623037764 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:27:22.140 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.152+0000 7fe798e5b640 1 -- 172.21.15.114:0/2623037764 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3154468949 0 0) 0x7fe78c02aba0 con 0x7fe78c006c70 2024-03-29T11:27:22.141 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.152+0000 7fe798e5b640 10 monclient(hunting): _negotiate 2024-03-29T11:27:22.141 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.152+0000 7fe798e5b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:22.141 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.152+0000 7fe798e5b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:22.141 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.152+0000 7fe798e5b640 10 monclient(hunting): my global_id is 4478 2024-03-29T11:27:22.141 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.152+0000 7fe798e5b640 1 -- 172.21.15.114:0/2623037764 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe780003520 con 0x7fe78c006c70 2024-03-29T11:27:22.142 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.154+0000 7fe798e5b640 1 -- 172.21.15.114:0/2623037764 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2383870004 0 0) 0x7fe78c02c830 con 0x7fe78c02b4c0 2024-03-29T11:27:22.142 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.154+0000 7fe798e5b640 10 monclient(hunting): _negotiate 2024-03-29T11:27:22.142 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.154+0000 7fe798e5b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:22.142 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.154+0000 7fe798e5b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:22.142 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.154+0000 7fe798e5b640 10 monclient(hunting): my global_id is 4464 2024-03-29T11:27:22.143 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.154+0000 7fe798e5b640 1 -- 172.21.15.114:0/2623037764 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe78c02aba0 con 0x7fe78c02b4c0 2024-03-29T11:27:22.143 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.155+0000 7fe798e5b640 1 -- 172.21.15.114:0/2623037764 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 543156267 0 0) 0x7fe78c02aba0 con 0x7fe78c02b4c0 2024-03-29T11:27:22.143 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.155+0000 7fe798e5b640 1 -- 172.21.15.114:0/2623037764 >> v1:172.21.15.114:6790/0 conn(0x7fe78c006c70 legacy=0x7fe78c0c6d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:22.143 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.155+0000 7fe798e5b640 1 -- 172.21.15.114:0/2623037764 >> v1:172.21.15.176:6789/0 conn(0x7fe78c006890 legacy=0x7fe78c0c0a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:27:22.143 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.155+0000 7fe798e5b640 10 monclient: _finish_hunting 0 2024-03-29T11:27:22.143 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.155+0000 7fe798e5b640 1 monclient: found mon.noname-a 2024-03-29T11:27:22.143 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.155+0000 7fe798e5b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:22.143 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.155+0000 7fe798e5b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.114:6789/0 2024-03-29T11:27:22.144 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.155+0000 7fe798e5b640 1 -- 172.21.15.114:0/2623037764 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe78c02ad30 con 0x7fe78c02b4c0 2024-03-29T11:27:22.144 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.155+0000 7fe798e5b640 10 monclient: _finish_auth 0 2024-03-29T11:27:22.144 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.155+0000 7fe798e5b640 10 monclient: _check_auth_tickets 2024-03-29T11:27:22.144 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.155+0000 7fe798e5b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:22.144 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.155+0000 7fe798e5b640 1 -- 172.21.15.114:0/2623037764 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fe784001670 con 0x7fe78c02b4c0 2024-03-29T11:27:22.144 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.155+0000 7fe798e5b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:22.144 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.155+0000 7fe798e5b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:27:22.144 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.155+0000 7fe798e5b640 10 monclient: dump: 2024-03-29T11:27:22.144 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:27:22.144 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:27:22.144 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:27:22.144 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:27:22.144 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:22.144 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:27:22.145 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:27:22.145 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:27:22.145 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:27:22.145 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:27:22.145 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.155+0000 7fe798e5b640 1 -- 172.21.15.114:0/2623037764 <== mon.0 v1:172.21.15.114:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe7840018d0 con 0x7fe78c02b4c0 2024-03-29T11:27:22.145 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.155+0000 7fe798e5b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:22.145 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.155+0000 7fe798e5b640 1 -- 172.21.15.114:0/2623037764 <== mon.0 v1:172.21.15.114:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fe784001670 con 0x7fe78c02b4c0 2024-03-29T11:27:22.145 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.155+0000 7fe798e5b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:22.145 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.155+0000 7fe798e5b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:27:22.145 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.155+0000 7fe798e5b640 10 monclient: dump: 2024-03-29T11:27:22.145 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:27:22.145 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:27:22.145 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:27:22.146 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:27:22.146 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:22.146 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:27:22.146 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:27:22.146 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:27:22.146 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:27:22.146 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:27:22.148 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.155+0000 7fe79ae5f640 5 monclient: authenticate success, global_id 4464 2024-03-29T11:27:22.148 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.155+0000 7fe79ae5f640 10 monclient: get_monmap_and_config success 2024-03-29T11:27:22.149 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.155+0000 7fe79ae5f640 4 set_mon_vals no callback set 2024-03-29T11:27:22.149 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.156+0000 7fe79ae5f640 10 monclient: shutdown 2024-03-29T11:27:22.149 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.156+0000 7fe79ae5f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:22.149 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.156+0000 7fe79ae5f640 1 -- 172.21.15.114:0/2623037764 >> v1:172.21.15.114:6789/0 conn(0x7fe78c02b4c0 legacy=0x7fe78c0cf0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:22.149 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.157+0000 7fe79ae5f640 1 -- 172.21.15.114:0/2623037764 shutdown_connections 2024-03-29T11:27:22.149 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.157+0000 7fe79ae5f640 1 -- 172.21.15.114:0/2623037764 >> 172.21.15.114:0/2623037764 conn(0x7fe78c0058f0 msgr2=0x7fe78c005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:22.149 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.157+0000 7fe79ae5f640 1 -- 172.21.15.114:0/2623037764 shutdown_connections 2024-03-29T11:27:22.149 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.158+0000 7fe79ae5f640 1 -- 172.21.15.114:0/2623037764 wait complete. 2024-03-29T11:27:22.149 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.158+0000 7fe79ae5f640 10 monclient: build_initial_monmap 2024-03-29T11:27:22.149 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.158+0000 7fe79ae5f640 10 monclient: monmap: 2024-03-29T11:27:22.149 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:27:22.149 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:27:22.149 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T11:27:22.149 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T11:27:22.150 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:22.150 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:27:22.150 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:27:22.150 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T11:27:22.150 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T11:27:22.150 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:27:22.150 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.159+0000 7fe79ae5f640 1 librados: starting msgr at 2024-03-29T11:27:22.150 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.159+0000 7fe79ae5f640 1 librados: starting objecter 2024-03-29T11:27:22.150 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.159+0000 7fe79ae5f640 1 Processor -- start 2024-03-29T11:27:22.150 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.159+0000 7fe79ae5f640 1 -- start start 2024-03-29T11:27:22.150 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.159+0000 7fe79ae5f640 1 librados: setting wanted keys 2024-03-29T11:27:22.151 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.159+0000 7fe79ae5f640 1 librados: calling monclient init 2024-03-29T11:27:22.151 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.159+0000 7fe79ae5f640 10 monclient: init 2024-03-29T11:27:22.151 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.160+0000 7fe79ae5f640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:22.152 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.161+0000 7fe79ae5f640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:27:22.153 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.163+0000 7fe79ae5f640 10 monclient(hunting): picked mon.noname-b con 0x7fe78c02b4c0 addr v1:172.21.15.176:6789/0 2024-03-29T11:27:22.153 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.164+0000 7fe799e5d640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7fe78c02b4c0 0x7fe78c0c3a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.114:55651/0 (socket says 172.21.15.114:55651) 2024-03-29T11:27:22.156 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.164+0000 7fe79ae5f640 10 monclient(hunting): picked mon.noname-a con 0x7fe78c0c3480 addr v1:172.21.15.114:6789/0 2024-03-29T11:27:22.156 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.164+0000 7fe79ae5f640 10 monclient(hunting): picked mon.noname-c con 0x7fe78c0bbf50 addr v1:172.21.15.114:6790/0 2024-03-29T11:27:22.156 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.164+0000 7fe79ae5f640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe78c0c4270 con 0x7fe78c0c3480 2024-03-29T11:27:22.157 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.164+0000 7fe79ae5f640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe78c0bf370 con 0x7fe78c02b4c0 2024-03-29T11:27:22.157 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.164+0000 7fe79ae5f640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe78c0bf500 con 0x7fe78c0bbf50 2024-03-29T11:27:22.157 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.164+0000 7fe79ae5f640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:22.157 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.164+0000 7fe79ae5f640 10 monclient(hunting): authenticate will time out at 1312.346899s 2024-03-29T11:27:22.157 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.165+0000 7fe799e5d640 1 -- 172.21.15.114:0/1138474631 learned_addr learned my addr 172.21.15.114:0/1138474631 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:27:22.157 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.165+0000 7fe78a7fc640 10 client.?.objecter ms_handle_connect 0x7fe78c02b4c0 2024-03-29T11:27:22.157 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.165+0000 7fe78a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:22.157 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.165+0000 7fe78a7fc640 1 -- 172.21.15.114:0/1138474631 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1418779700 0 0) 0x7fe78c0bf370 con 0x7fe78c02b4c0 2024-03-29T11:27:22.158 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.165+0000 7fe78a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:27:22.158 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.165+0000 7fe78a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:22.158 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.165+0000 7fe78a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:22.158 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.165+0000 7fe78a7fc640 10 monclient(hunting): my global_id is 4438 2024-03-29T11:27:22.158 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.165+0000 7fe78a7fc640 1 -- 172.21.15.114:0/1138474631 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe778004eb0 con 0x7fe78c02b4c0 2024-03-29T11:27:22.158 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.165+0000 7fe78a7fc640 1 -- 172.21.15.114:0/1138474631 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2477293698 0 0) 0x7fe778004eb0 con 0x7fe78c02b4c0 2024-03-29T11:27:22.158 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.165+0000 7fe78a7fc640 1 -- 172.21.15.114:0/1138474631 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe78c0bf370 con 0x7fe78c02b4c0 2024-03-29T11:27:22.158 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.166+0000 7fe78a7fc640 1 -- 172.21.15.114:0/1138474631 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fe7900018b0 con 0x7fe78c02b4c0 2024-03-29T11:27:22.158 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.166+0000 7fe78a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:22.158 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.166+0000 7fe78a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:27:22.158 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.166+0000 7fe78a7fc640 10 monclient(hunting): dump: 2024-03-29T11:27:22.158 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:27:22.158 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:27:22.158 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:27:22.159 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:27:22.159 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:22.159 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:27:22.159 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:27:22.159 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:27:22.159 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:27:22.159 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:27:22.159 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.166+0000 7fe78a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:27:22.159 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.166+0000 7fe78a7fc640 1 -- 172.21.15.114:0/1138474631 <== mon.1 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4236057613 0 0) 0x7fe78c0bf370 con 0x7fe78c02b4c0 2024-03-29T11:27:22.160 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.166+0000 7fe78a7fc640 1 -- 172.21.15.114:0/1138474631 >> v1:172.21.15.114:6790/0 conn(0x7fe78c0bbf50 legacy=0x7fe78c160cd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:27:22.160 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.166+0000 7fe78a7fc640 1 -- 172.21.15.114:0/1138474631 >> v1:172.21.15.114:6789/0 conn(0x7fe78c0c3480 legacy=0x7fe78c0bb840 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:27:22.160 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.166+0000 7fe78a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:27:22.160 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.166+0000 7fe78a7fc640 1 monclient: found mon.b 2024-03-29T11:27:22.160 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.166+0000 7fe78a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:22.160 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.166+0000 7fe78a7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:27:22.160 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.167+0000 7fe78a7fc640 1 -- 172.21.15.114:0/1138474631 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe78c1613e0 con 0x7fe78c02b4c0 2024-03-29T11:27:22.160 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.167+0000 7fe78a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:27:22.160 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.167+0000 7fe78a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:27:22.160 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.167+0000 7fe78a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:22.160 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.167+0000 7fe79ae5f640 5 monclient: authenticate success, global_id 4438 2024-03-29T11:27:22.161 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.167+0000 7fe79ae5f640 10 monclient: _renew_subs 2024-03-29T11:27:22.161 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.167+0000 7fe79ae5f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:27:22.161 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.167+0000 7fe78a7fc640 1 -- 172.21.15.114:0/1138474631 <== mon.1 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe790001d10 con 0x7fe78c02b4c0 2024-03-29T11:27:22.161 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.167+0000 7fe79ae5f640 1 -- 172.21.15.114:0/1138474631 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe78c1616b0 con 0x7fe78c02b4c0 2024-03-29T11:27:22.161 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.167+0000 7fe79ae5f640 10 client.4438.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:27:22.161 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.167+0000 7fe79ae5f640 10 monclient: _renew_subs 2024-03-29T11:27:22.161 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.167+0000 7fe79ae5f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:27:22.161 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.167+0000 7fe79ae5f640 1 -- 172.21.15.114:0/1138474631 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe78c16d7e0 con 0x7fe78c02b4c0 2024-03-29T11:27:22.161 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.167+0000 7fe79ae5f640 1 librados: init done 2024-03-29T11:27:22.162 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.167+0000 7fe78a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:22.162 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.167+0000 7fe78a7fc640 1 -- 172.21.15.114:0/1138474631 <== mon.1 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fe790003a40 con 0x7fe78c02b4c0 2024-03-29T11:27:22.162 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.168+0000 7fe78a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:22.162 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.168+0000 7fe78a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:27:22.162 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.168+0000 7fe78bfff640 4 set_mon_vals no callback set 2024-03-29T11:27:22.162 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.168+0000 7fe78a7fc640 10 monclient: dump: 2024-03-29T11:27:22.162 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:27:22.162 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:27:22.162 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:27:22.162 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:27:22.163 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:22.163 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:27:22.163 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:27:22.163 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:27:22.163 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:27:22.163 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:27:22.165 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.170+0000 7fe79ae5f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:22.165 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.170+0000 7fe79ae5f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:22.165 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.170+0000 7fe79ae5f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:27:22.165 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.170+0000 7fe79ae5f640 1 -- 172.21.15.114:0/1138474631 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe77c005080 con 0x7fe78c02b4c0 2024-03-29T11:27:22.165 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.173+0000 7fe78a7fc640 1 -- 172.21.15.114:0/1138474631 <== mon.1 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1594292461 0 0) 0x7fe790003ba0 con 0x7fe78c02b4c0 2024-03-29T11:27:22.165 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.174+0000 7fe79965c640 10 monclient: get_auth_request con 0x7fe77806dc10 auth_method 0 2024-03-29T11:27:22.165 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.174+0000 7fe78a7fc640 10 client.4438.objecter ms_handle_connect 0x7fe77806dc10 2024-03-29T11:27:22.165 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.174+0000 7fe78a7fc640 1 -- 172.21.15.114:0/1138474631 <== mon.1 v1:172.21.15.176:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3238987313 0 0) 0x7fe790084a90 con 0x7fe78c02b4c0 2024-03-29T11:27:22.165 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.174+0000 7fe78a7fc640 10 client.4438.objecter ms_dispatch 0x7fe794000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:27:22.165 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.175+0000 7fe78a7fc640 3 client.4438.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:27:22.165 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.175+0000 7fe78a7fc640 3 client.4438.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:27:22.165 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.175+0000 7fe78a7fc640 20 client.4438.objecter dump_active .. 0 homeless 2024-03-29T11:27:22.168 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.180+0000 7fe78a7fc640 1 -- 172.21.15.114:0/1138474631 <== mon.1 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe790050370 con 0x7fe78c02b4c0 2024-03-29T11:27:22.168 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.180+0000 7fe78a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:22.168 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.180+0000 7fe78a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:27:22.498 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.510+0000 7fe79ae5f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:22.498 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.510+0000 7fe79ae5f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:22.498 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.510+0000 7fe79ae5f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:27:22.499 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.510+0000 7fe79ae5f640 1 -- 172.21.15.114:0/1138474631 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe77c005330 con 0x7fe78c02b4c0 2024-03-29T11:27:22.501 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.513+0000 7fe78a7fc640 1 -- 172.21.15.114:0/1138474631 <== mon.1 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2858224613) 0x7fe7900522a0 con 0x7fe78c02b4c0 2024-03-29T11:27:22.501 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.513+0000 7fe78a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:22.501 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.513+0000 7fe78a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:27:22.502 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T11:27:22.502 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"b0de2b9b-c45a-44c7-b6ee-e83b39a1e108","created":"2024-03-29T11:20:49.420660+0000","modified":"2024-03-29T11:20:51.700167+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:51.700167+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a64785d3-6b11-499a-b706-cc827ab96a74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5882dadb-e7a8-4604-ba9a-418013c106c8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d7e8c409-bc66-4e4a-bbdb-420bad31388e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0682989-3595-4cd6-90a5-ab9111c278e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"67068120-e3ff-4817-bd06-44feb7479a35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09e8b-8a85-4997-bdea-75d754e6ee67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2f0d9ead-f47f-4cb3-92a4-0a5ad23cd831","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fdb6a25f-6bd8-4c65-9b9b-b460ba7673a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:27:22.504 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.516+0000 7fe79ae5f640 10 librados: watch_flush enter 2024-03-29T11:27:22.504 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.516+0000 7fe79ae5f640 10 librados: watch_flush exit 2024-03-29T11:27:22.504 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.516+0000 7fe79ae5f640 20 client.4438.objecter shutdown clearing up homeless session... 2024-03-29T11:27:22.505 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.516+0000 7fe79ae5f640 10 client.4438.objecter successfully canceled tick 2024-03-29T11:27:22.505 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.516+0000 7fe79ae5f640 1 -- 172.21.15.114:0/1138474631 >> v1:172.21.15.114:6816/4038755055 conn(0x7fe77806dc10 legacy=0x7fe7780700d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:22.505 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.516+0000 7fe79ae5f640 10 monclient: shutdown 2024-03-29T11:27:22.505 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.516+0000 7fe79ae5f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:22.505 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.516+0000 7fe79ae5f640 1 -- 172.21.15.114:0/1138474631 >> v1:172.21.15.176:6789/0 conn(0x7fe78c02b4c0 legacy=0x7fe78c0c3a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:22.505 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.517+0000 7fe79ae5f640 1 -- 172.21.15.114:0/1138474631 shutdown_connections 2024-03-29T11:27:22.505 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.517+0000 7fe79ae5f640 1 -- 172.21.15.114:0/1138474631 >> 172.21.15.114:0/1138474631 conn(0x7fe78c0058f0 msgr2=0x7fe78c038130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:22.506 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.518+0000 7fe79ae5f640 1 -- 172.21.15.114:0/1138474631 shutdown_connections 2024-03-29T11:27:22.506 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.518+0000 7fe79ae5f640 1 -- 172.21.15.114:0/1138474631 wait complete. 2024-03-29T11:27:22.506 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:22.518+0000 7fe79ae5f640 1 librados: shutdown 2024-03-29T11:27:22.525 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:27:29.332 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:27:29.472 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.484+0000 7fbbf2a64640 10 monclient: get_monmap_and_config 2024-03-29T11:27:29.472 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.484+0000 7fbbf2a64640 10 monclient: build_initial_monmap 2024-03-29T11:27:29.472 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.484+0000 7fbbf2a64640 10 monclient: monmap: 2024-03-29T11:27:29.473 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:27:29.473 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:27:29.473 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:27:29.485572+0000 2024-03-29T11:27:29.473 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:27:29.485572+0000 2024-03-29T11:27:29.473 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:29.473 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:27:29.473 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:27:29.473 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T11:27:29.473 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T11:27:29.473 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:27:29.474 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.486+0000 7fbbf2a64640 1 Processor -- start 2024-03-29T11:27:29.474 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.486+0000 7fbbf2a64640 1 -- start start 2024-03-29T11:27:29.474 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.486+0000 7fbbf2a64640 10 monclient: init 2024-03-29T11:27:29.474 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.487+0000 7fbbf2a64640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:29.475 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.487+0000 7fbbf2a64640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:27:29.475 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.487+0000 7fbbf2a64640 10 monclient(hunting): picked mon.noname-b con 0x7fbbec0b46b0 addr v1:172.21.15.114:6790/0 2024-03-29T11:27:29.475 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.487+0000 7fbbf2a64640 10 monclient(hunting): picked mon.noname-a con 0x7fbbec0aeae0 addr v1:172.21.15.114:6789/0 2024-03-29T11:27:29.476 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.487+0000 7fbbf2a64640 10 monclient(hunting): picked mon.noname-c con 0x7fbbec13fa10 addr v1:172.21.15.176:6789/0 2024-03-29T11:27:29.476 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.487+0000 7fbbf2a64640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbec0b06a0 con 0x7fbbec0aeae0 2024-03-29T11:27:29.476 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.487+0000 7fbbf2a64640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbec0b4230 con 0x7fbbec13fa10 2024-03-29T11:27:29.476 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.487+0000 7fbbf2a64640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbec0b0910 con 0x7fbbec0b46b0 2024-03-29T11:27:29.476 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.487+0000 7fbbf2a64640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:29.476 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.487+0000 7fbbf2a64640 10 monclient(hunting): authenticate will time out at 1319.669350s 2024-03-29T11:27:29.476 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.487+0000 7fbbf0800640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7fbbec0b46b0 0x7fbbec0b3b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:36330/0 (socket says 172.21.15.114:36330) 2024-03-29T11:27:29.476 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.487+0000 7fbbf0800640 1 -- 172.21.15.114:0/3507400100 learned_addr learned my addr 172.21.15.114:0/3507400100 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:27:29.476 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.487+0000 7fbbeb7fe640 1 -- 172.21.15.114:0/3507400100 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 144730241 0 0) 0x7fbbec0b06a0 con 0x7fbbec0aeae0 2024-03-29T11:27:29.476 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.487+0000 7fbbeb7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:27:29.476 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.487+0000 7fbbeb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:29.476 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.487+0000 7fbbeb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:29.476 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.487+0000 7fbbeb7fe640 10 monclient(hunting): my global_id is 4467 2024-03-29T11:27:29.476 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.487+0000 7fbbeb7fe640 1 -- 172.21.15.114:0/3507400100 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbbd8003520 con 0x7fbbec0aeae0 2024-03-29T11:27:29.477 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.487+0000 7fbbeb7fe640 1 -- 172.21.15.114:0/3507400100 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2084726690 0 0) 0x7fbbec0b0910 con 0x7fbbec0b46b0 2024-03-29T11:27:29.477 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.487+0000 7fbbeb7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:27:29.477 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.487+0000 7fbbeb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:29.477 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.487+0000 7fbbeb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:29.477 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.487+0000 7fbbeb7fe640 10 monclient(hunting): my global_id is 4481 2024-03-29T11:27:29.477 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.487+0000 7fbbeb7fe640 1 -- 172.21.15.114:0/3507400100 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbbec0b06a0 con 0x7fbbec0b46b0 2024-03-29T11:27:29.477 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.487+0000 7fbbeb7fe640 1 -- 172.21.15.114:0/3507400100 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2711775651 0 0) 0x7fbbd8003520 con 0x7fbbec0aeae0 2024-03-29T11:27:29.477 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.487+0000 7fbbeb7fe640 1 -- 172.21.15.114:0/3507400100 >> v1:172.21.15.114:6790/0 conn(0x7fbbec0b46b0 legacy=0x7fbbec0b3b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:29.477 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.487+0000 7fbbeb7fe640 1 -- 172.21.15.114:0/3507400100 >> v1:172.21.15.176:6789/0 conn(0x7fbbec13fa10 legacy=0x7fbbec14d9e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:27:29.478 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.490+0000 7fbbeb7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:27:29.478 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.490+0000 7fbbeb7fe640 1 monclient: found mon.noname-a 2024-03-29T11:27:29.478 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.490+0000 7fbbeb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:29.478 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.490+0000 7fbbeb7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.114:6789/0 2024-03-29T11:27:29.478 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.490+0000 7fbbeb7fe640 1 -- 172.21.15.114:0/3507400100 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbec0af6d0 con 0x7fbbec0aeae0 2024-03-29T11:27:29.479 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.490+0000 7fbbeb7fe640 10 monclient: _finish_auth 0 2024-03-29T11:27:29.479 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.490+0000 7fbbeb7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:27:29.479 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.490+0000 7fbbeb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:29.479 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.490+0000 7fbbeb7fe640 1 -- 172.21.15.114:0/3507400100 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fbbe4001690 con 0x7fbbec0aeae0 2024-03-29T11:27:29.479 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.490+0000 7fbbeb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:29.479 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.490+0000 7fbbeb7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:27:29.479 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.490+0000 7fbbeb7fe640 10 monclient: dump: 2024-03-29T11:27:29.479 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:27:29.479 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:27:29.479 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:27:29.479 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:27:29.479 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:29.479 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:27:29.479 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:27:29.480 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:27:29.480 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:27:29.480 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:27:29.480 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.490+0000 7fbbf2a64640 5 monclient: authenticate success, global_id 4467 2024-03-29T11:27:29.480 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.490+0000 7fbbf2a64640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:29.480 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.490+0000 7fbbeb7fe640 1 -- 172.21.15.114:0/3507400100 <== mon.0 v1:172.21.15.114:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbbe40017f0 con 0x7fbbec0aeae0 2024-03-29T11:27:29.480 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.490+0000 7fbbeb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:29.480 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.490+0000 7fbbeb7fe640 1 -- 172.21.15.114:0/3507400100 <== mon.0 v1:172.21.15.114:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fbbe4001c00 con 0x7fbbec0aeae0 2024-03-29T11:27:29.480 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.490+0000 7fbbf2a64640 10 monclient: get_monmap_and_config success 2024-03-29T11:27:29.480 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.490+0000 7fbbeb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:29.480 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.490+0000 7fbbeb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:27:29.480 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.490+0000 7fbbeb7fe640 10 monclient: dump: 2024-03-29T11:27:29.480 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:27:29.480 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:27:29.480 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:27:29.481 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:27:29.481 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:29.481 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:27:29.481 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:27:29.481 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:27:29.481 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:27:29.481 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:27:29.481 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.490+0000 7fbbf2a64640 4 set_mon_vals no callback set 2024-03-29T11:27:29.488 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.491+0000 7fbbf2a64640 10 monclient: shutdown 2024-03-29T11:27:29.489 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.491+0000 7fbbf2a64640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:29.489 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.491+0000 7fbbf2a64640 1 -- 172.21.15.114:0/3507400100 >> v1:172.21.15.114:6789/0 conn(0x7fbbec0aeae0 legacy=0x7fbbec13f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:29.489 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.491+0000 7fbbf2a64640 1 -- 172.21.15.114:0/3507400100 shutdown_connections 2024-03-29T11:27:29.489 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.491+0000 7fbbf2a64640 1 -- 172.21.15.114:0/3507400100 >> 172.21.15.114:0/3507400100 conn(0x7fbbec066a00 msgr2=0x7fbbec066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:29.489 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.491+0000 7fbbf2a64640 1 -- 172.21.15.114:0/3507400100 shutdown_connections 2024-03-29T11:27:29.489 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.491+0000 7fbbf2a64640 1 -- 172.21.15.114:0/3507400100 wait complete. 2024-03-29T11:27:29.489 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.492+0000 7fbbf2a64640 10 monclient: build_initial_monmap 2024-03-29T11:27:29.489 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.492+0000 7fbbf2a64640 10 monclient: monmap: 2024-03-29T11:27:29.489 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:27:29.489 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:27:29.489 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T11:27:29.490 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T11:27:29.490 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:29.490 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:27:29.490 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:27:29.490 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T11:27:29.490 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T11:27:29.490 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:27:29.490 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.492+0000 7fbbf2a64640 1 librados: starting msgr at 2024-03-29T11:27:29.490 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.492+0000 7fbbf2a64640 1 librados: starting objecter 2024-03-29T11:27:29.490 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.492+0000 7fbbf2a64640 1 Processor -- start 2024-03-29T11:27:29.490 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.492+0000 7fbbf2a64640 1 -- start start 2024-03-29T11:27:29.490 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.492+0000 7fbbf2a64640 1 librados: setting wanted keys 2024-03-29T11:27:29.490 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.492+0000 7fbbf2a64640 1 librados: calling monclient init 2024-03-29T11:27:29.491 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.492+0000 7fbbf2a64640 10 monclient: init 2024-03-29T11:27:29.491 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.493+0000 7fbbf2a64640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:29.491 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.493+0000 7fbbf2a64640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:27:29.491 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.493+0000 7fbbf2a64640 10 monclient(hunting): picked mon.noname-a con 0x7fbbec0aeae0 addr v1:172.21.15.114:6789/0 2024-03-29T11:27:29.491 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.493+0000 7fbbf2a64640 10 monclient(hunting): picked mon.noname-c con 0x7fbbec0b46b0 addr v1:172.21.15.114:6790/0 2024-03-29T11:27:29.491 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.494+0000 7fbbf0800640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fbbec0aeae0 0x7fbbec1449a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:38146/0 (socket says 172.21.15.114:38146) 2024-03-29T11:27:29.491 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.494+0000 7fbbebfff640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7fbbec0b46b0 0x7fbbec0b5c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:36668/0 (socket says 172.21.15.114:36668) 2024-03-29T11:27:29.491 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.494+0000 7fbbf2a64640 10 monclient(hunting): picked mon.noname-b con 0x7fbbec13fa10 addr v1:172.21.15.176:6789/0 2024-03-29T11:27:29.491 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.494+0000 7fbbf0800640 1 -- 172.21.15.114:0/4113991244 learned_addr learned my addr 172.21.15.114:0/4113991244 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:27:29.491 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.494+0000 7fbbe97fa640 10 client.?.objecter ms_handle_connect 0x7fbbec0b46b0 2024-03-29T11:27:29.491 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.494+0000 7fbbe97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:29.491 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.494+0000 7fbbf2a64640 1 -- 172.21.15.114:0/4113991244 --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbec0b7300 con 0x7fbbec0aeae0 2024-03-29T11:27:29.491 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.494+0000 7fbbf2a64640 1 -- 172.21.15.114:0/4113991244 --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbec145e10 con 0x7fbbec13fa10 2024-03-29T11:27:29.491 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.494+0000 7fbbf2a64640 1 -- 172.21.15.114:0/4113991244 --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbec1460f0 con 0x7fbbec0b46b0 2024-03-29T11:27:29.492 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.494+0000 7fbbf2a64640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:29.492 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.494+0000 7fbbf2a64640 10 monclient(hunting): authenticate will time out at 1319.677050s 2024-03-29T11:27:29.492 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.495+0000 7fbbe97fa640 1 -- 172.21.15.114:0/4113991244 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3395505066 0 0) 0x7fbbec1460f0 con 0x7fbbec0b46b0 2024-03-29T11:27:29.492 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.495+0000 7fbbe97fa640 10 monclient(hunting): _negotiate 2024-03-29T11:27:29.492 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.495+0000 7fbbe97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:29.492 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.495+0000 7fbbe97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:29.492 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.495+0000 7fbbe97fa640 10 monclient(hunting): my global_id is 4484 2024-03-29T11:27:29.492 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.495+0000 7fbbe97fa640 1 -- 172.21.15.114:0/4113991244 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbbcc004eb0 con 0x7fbbec0b46b0 2024-03-29T11:27:29.492 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.495+0000 7fbbe97fa640 10 client.?.objecter ms_handle_connect 0x7fbbec0aeae0 2024-03-29T11:27:29.492 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.495+0000 7fbbe97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:29.492 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.495+0000 7fbbe97fa640 1 -- 172.21.15.114:0/4113991244 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2520574221 0 0) 0x7fbbcc004eb0 con 0x7fbbec0b46b0 2024-03-29T11:27:29.492 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.495+0000 7fbbe97fa640 1 -- 172.21.15.114:0/4113991244 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbbec1460f0 con 0x7fbbec0b46b0 2024-03-29T11:27:29.492 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.495+0000 7fbbe97fa640 1 -- 172.21.15.114:0/4113991244 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fbbe4001bc0 con 0x7fbbec0b46b0 2024-03-29T11:27:29.492 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.495+0000 7fbbe97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:29.492 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.495+0000 7fbbe97fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:27:29.493 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.495+0000 7fbbe97fa640 10 monclient(hunting): dump: 2024-03-29T11:27:29.493 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:27:29.493 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:27:29.493 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:27:29.493 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:27:29.493 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:29.493 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:27:29.493 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:27:29.493 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:27:29.493 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:27:29.493 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:27:29.493 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.495+0000 7fbbe97fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:27:29.493 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.495+0000 7fbbe97fa640 1 -- 172.21.15.114:0/4113991244 <== mon.2 v1:172.21.15.114:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 468132922 0 0) 0x7fbbec1460f0 con 0x7fbbec0b46b0 2024-03-29T11:27:29.493 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.495+0000 7fbbe97fa640 1 -- 172.21.15.114:0/4113991244 >> v1:172.21.15.176:6789/0 conn(0x7fbbec13fa10 legacy=0x7fbbec141870 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:27:29.493 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.496+0000 7fbbe97fa640 1 -- 172.21.15.114:0/4113991244 >> v1:172.21.15.114:6789/0 conn(0x7fbbec0aeae0 legacy=0x7fbbec1449a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:29.494 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.496+0000 7fbbe97fa640 10 monclient: _finish_hunting 0 2024-03-29T11:27:29.494 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.496+0000 7fbbe97fa640 1 monclient: found mon.c 2024-03-29T11:27:29.494 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.496+0000 7fbbe97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:29.494 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.496+0000 7fbbe97fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:27:29.494 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.496+0000 7fbbe97fa640 1 -- 172.21.15.114:0/4113991244 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbec0b6350 con 0x7fbbec0b46b0 2024-03-29T11:27:29.494 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.496+0000 7fbbe97fa640 10 monclient: _finish_auth 0 2024-03-29T11:27:29.494 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.496+0000 7fbbe97fa640 10 monclient: _check_auth_tickets 2024-03-29T11:27:29.494 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.496+0000 7fbbe97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:29.494 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.496+0000 7fbbf2a64640 5 monclient: authenticate success, global_id 4484 2024-03-29T11:27:29.494 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.496+0000 7fbbf2a64640 10 monclient: _renew_subs 2024-03-29T11:27:29.494 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.496+0000 7fbbf2a64640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:27:29.494 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.496+0000 7fbbf2a64640 1 -- 172.21.15.114:0/4113991244 --> v1:172.21.15.114:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbec145390 con 0x7fbbec0b46b0 2024-03-29T11:27:29.494 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.496+0000 7fbbf2a64640 10 client.4484.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:27:29.494 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.496+0000 7fbbf2a64640 10 monclient: _renew_subs 2024-03-29T11:27:29.495 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.496+0000 7fbbf2a64640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:27:29.495 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.496+0000 7fbbf2a64640 1 -- 172.21.15.114:0/4113991244 --> v1:172.21.15.114:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbec144230 con 0x7fbbec0b46b0 2024-03-29T11:27:29.495 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.496+0000 7fbbf2a64640 1 librados: init done 2024-03-29T11:27:29.495 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.497+0000 7fbbe97fa640 1 -- 172.21.15.114:0/4113991244 <== mon.2 v1:172.21.15.114:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbbe40038a0 con 0x7fbbec0b46b0 2024-03-29T11:27:29.495 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.497+0000 7fbbe97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:29.495 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.497+0000 7fbbeaffd640 4 set_mon_vals no callback set 2024-03-29T11:27:29.495 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.498+0000 7fbbe97fa640 1 -- 172.21.15.114:0/4113991244 <== mon.2 v1:172.21.15.114:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fbbe4003cb0 con 0x7fbbec0b46b0 2024-03-29T11:27:29.495 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.498+0000 7fbbe97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:29.495 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.498+0000 7fbbe97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:29.496 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.498+0000 7fbbe97fa640 10 monclient: dump: 2024-03-29T11:27:29.496 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:27:29.496 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:27:29.496 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:27:29.496 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:27:29.496 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:29.496 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:27:29.496 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:27:29.496 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:27:29.496 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:27:29.496 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:27:29.496 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.498+0000 7fbbf2a64640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:29.496 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.499+0000 7fbbf2a64640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:29.496 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.499+0000 7fbbf2a64640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:27:29.497 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.499+0000 7fbbf2a64640 1 -- 172.21.15.114:0/4113991244 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbd0005080 con 0x7fbbec0b46b0 2024-03-29T11:27:29.497 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.499+0000 7fbbe97fa640 1 -- 172.21.15.114:0/4113991244 <== mon.2 v1:172.21.15.114:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1594292461 0 0) 0x7fbbe4003ed0 con 0x7fbbec0b46b0 2024-03-29T11:27:29.497 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.500+0000 7fbbe97fa640 1 -- 172.21.15.114:0/4113991244 <== mon.2 v1:172.21.15.114:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3238987313 0 0) 0x7fbbe4084b70 con 0x7fbbec0b46b0 2024-03-29T11:27:29.497 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.500+0000 7fbbe97fa640 10 client.4484.objecter ms_dispatch 0x7fbbec000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:27:29.497 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.500+0000 7fbbe97fa640 3 client.4484.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:27:29.497 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.500+0000 7fbbe97fa640 3 client.4484.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:27:29.497 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.500+0000 7fbbf0800640 10 monclient: get_auth_request con 0x7fbbcc06dbb0 auth_method 0 2024-03-29T11:27:29.497 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.500+0000 7fbbe97fa640 20 client.4484.objecter dump_active .. 0 homeless 2024-03-29T11:27:29.497 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.500+0000 7fbbe97fa640 10 client.4484.objecter ms_handle_connect 0x7fbbcc06dbb0 2024-03-29T11:27:29.498 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.507+0000 7fbbe97fa640 1 -- 172.21.15.114:0/4113991244 <== mon.2 v1:172.21.15.114:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbbe4050450 con 0x7fbbec0b46b0 2024-03-29T11:27:29.498 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.507+0000 7fbbe97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:29.498 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.507+0000 7fbbe97fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:27:29.809 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.820+0000 7fbbf2a64640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:29.809 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.820+0000 7fbbf2a64640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:29.809 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.820+0000 7fbbf2a64640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:27:29.809 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.820+0000 7fbbf2a64640 1 -- 172.21.15.114:0/4113991244 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbbd0005330 con 0x7fbbec0b46b0 2024-03-29T11:27:29.810 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.822+0000 7fbbe97fa640 1 -- 172.21.15.114:0/4113991244 <== mon.2 v1:172.21.15.114:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2858224613) 0x7fbbe4052380 con 0x7fbbec0b46b0 2024-03-29T11:27:29.810 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.822+0000 7fbbe97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:29.810 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.822+0000 7fbbe97fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:27:29.810 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T11:27:29.810 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"b0de2b9b-c45a-44c7-b6ee-e83b39a1e108","created":"2024-03-29T11:20:49.420660+0000","modified":"2024-03-29T11:20:51.700167+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:51.700167+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a64785d3-6b11-499a-b706-cc827ab96a74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5882dadb-e7a8-4604-ba9a-418013c106c8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d7e8c409-bc66-4e4a-bbdb-420bad31388e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0682989-3595-4cd6-90a5-ab9111c278e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"67068120-e3ff-4817-bd06-44feb7479a35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09e8b-8a85-4997-bdea-75d754e6ee67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2f0d9ead-f47f-4cb3-92a4-0a5ad23cd831","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fdb6a25f-6bd8-4c65-9b9b-b460ba7673a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:27:29.813 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.825+0000 7fbbf2a64640 10 librados: watch_flush enter 2024-03-29T11:27:29.813 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.825+0000 7fbbf2a64640 10 librados: watch_flush exit 2024-03-29T11:27:29.813 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.825+0000 7fbbf2a64640 20 client.4484.objecter shutdown clearing up homeless session... 2024-03-29T11:27:29.813 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.825+0000 7fbbf2a64640 10 client.4484.objecter successfully canceled tick 2024-03-29T11:27:29.814 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.826+0000 7fbbf2a64640 1 -- 172.21.15.114:0/4113991244 >> v1:172.21.15.114:6816/4038755055 conn(0x7fbbcc06dbb0 legacy=0x7fbbcc070070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:29.814 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.826+0000 7fbbf2a64640 10 monclient: shutdown 2024-03-29T11:27:29.814 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.826+0000 7fbbf2a64640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:29.815 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.826+0000 7fbbf2a64640 1 -- 172.21.15.114:0/4113991244 >> v1:172.21.15.114:6790/0 conn(0x7fbbec0b46b0 legacy=0x7fbbec0b5c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:29.815 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.826+0000 7fbbf2a64640 1 -- 172.21.15.114:0/4113991244 shutdown_connections 2024-03-29T11:27:29.815 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.826+0000 7fbbf2a64640 1 -- 172.21.15.114:0/4113991244 >> 172.21.15.114:0/4113991244 conn(0x7fbbec066a00 msgr2=0x7fbbec13f0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:29.815 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.826+0000 7fbbf2a64640 1 -- 172.21.15.114:0/4113991244 shutdown_connections 2024-03-29T11:27:29.815 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.826+0000 7fbbf2a64640 1 -- 172.21.15.114:0/4113991244 wait complete. 2024-03-29T11:27:29.815 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:29.826+0000 7fbbf2a64640 1 librados: shutdown 2024-03-29T11:27:29.828 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:27:36.634 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:27:36.795 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.806+0000 7f1897c06640 10 monclient: get_monmap_and_config 2024-03-29T11:27:36.795 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.806+0000 7f1897c06640 10 monclient: build_initial_monmap 2024-03-29T11:27:36.795 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.806+0000 7f1897c06640 10 monclient: monmap: 2024-03-29T11:27:36.795 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:27:36.795 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:27:36.796 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:27:36.807132+0000 2024-03-29T11:27:36.796 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:27:36.807132+0000 2024-03-29T11:27:36.796 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:36.796 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:27:36.796 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:27:36.796 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T11:27:36.796 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T11:27:36.796 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:27:36.798 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.807+0000 7f1897c06640 1 Processor -- start 2024-03-29T11:27:36.798 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.807+0000 7f1897c06640 1 -- start start 2024-03-29T11:27:36.798 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.807+0000 7f1897c06640 10 monclient: init 2024-03-29T11:27:36.798 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.808+0000 7f1897c06640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:36.799 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.808+0000 7f1897c06640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:27:36.799 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.808+0000 7f1897c06640 10 monclient(hunting): picked mon.noname-b con 0x7f18900a9930 addr v1:172.21.15.114:6790/0 2024-03-29T11:27:36.799 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.808+0000 7f1897c06640 10 monclient(hunting): picked mon.noname-c con 0x7f1890149dc0 addr v1:172.21.15.176:6789/0 2024-03-29T11:27:36.799 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.808+0000 7f1897c06640 10 monclient(hunting): picked mon.noname-a con 0x7f189014ff40 addr v1:172.21.15.114:6789/0 2024-03-29T11:27:36.799 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.808+0000 7f1897c06640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18900aae60 con 0x7f189014ff40 2024-03-29T11:27:36.799 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.808+0000 7f1897c06640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18900b4310 con 0x7f1890149dc0 2024-03-29T11:27:36.800 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.808+0000 7f1897c06640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18900afdd0 con 0x7f18900a9930 2024-03-29T11:27:36.800 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.808+0000 7f1897c06640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:36.800 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.808+0000 7f1897c06640 10 monclient(hunting): authenticate will time out at 1326.990513s 2024-03-29T11:27:36.800 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.809+0000 7f18951a1640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f1890149dc0 0x7f189014a240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.114:52676/0 (socket says 172.21.15.114:52676) 2024-03-29T11:27:36.800 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.809+0000 7f18951a1640 1 -- 172.21.15.114:0/1472722014 learned_addr learned my addr 172.21.15.114:0/1472722014 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:27:36.800 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.810+0000 7f18949a0640 1 -- 172.21.15.114:0/1472722014 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3707596162 0 0) 0x7f18900b4310 con 0x7f1890149dc0 2024-03-29T11:27:36.800 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.810+0000 7f18949a0640 10 monclient(hunting): _negotiate 2024-03-29T11:27:36.800 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.810+0000 7f18949a0640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:36.800 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.810+0000 7f18949a0640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:36.800 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.810+0000 7f18949a0640 10 monclient(hunting): my global_id is 4441 2024-03-29T11:27:36.800 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.810+0000 7f18949a0640 1 -- 172.21.15.114:0/1472722014 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f187c003520 con 0x7f1890149dc0 2024-03-29T11:27:36.802 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.812+0000 7f18949a0640 1 -- 172.21.15.114:0/1472722014 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1472267829 0 0) 0x7f187c003520 con 0x7f1890149dc0 2024-03-29T11:27:36.802 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.812+0000 7f18949a0640 1 -- 172.21.15.114:0/1472722014 >> v1:172.21.15.114:6790/0 conn(0x7f18900a9930 legacy=0x7f1890143a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:27:36.802 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.812+0000 7f18949a0640 1 -- 172.21.15.114:0/1472722014 >> v1:172.21.15.114:6789/0 conn(0x7f189014ff40 legacy=0x7f1890152330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:27:36.802 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.812+0000 7f18949a0640 10 monclient: _finish_hunting 0 2024-03-29T11:27:36.802 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.812+0000 7f18949a0640 1 monclient: found mon.noname-c 2024-03-29T11:27:36.802 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.812+0000 7f18949a0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:36.803 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.812+0000 7f18949a0640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.176:6789/0 2024-03-29T11:27:36.803 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.812+0000 7f18949a0640 1 -- 172.21.15.114:0/1472722014 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f189014a950 con 0x7f1890149dc0 2024-03-29T11:27:36.803 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.812+0000 7f18949a0640 10 monclient: _finish_auth 0 2024-03-29T11:27:36.803 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.812+0000 7f18949a0640 10 monclient: _check_auth_tickets 2024-03-29T11:27:36.803 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.812+0000 7f18949a0640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:36.803 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.812+0000 7f18949a0640 1 -- 172.21.15.114:0/1472722014 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f1880001690 con 0x7f1890149dc0 2024-03-29T11:27:36.804 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.812+0000 7f18949a0640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:36.804 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.812+0000 7f18949a0640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:27:36.804 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.812+0000 7f18949a0640 10 monclient: dump: 2024-03-29T11:27:36.804 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:27:36.804 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:27:36.804 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:27:36.804 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:27:36.804 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:36.804 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:27:36.804 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:27:36.804 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:27:36.804 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:27:36.804 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:27:36.804 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.812+0000 7f1897c06640 5 monclient: authenticate success, global_id 4441 2024-03-29T11:27:36.804 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.812+0000 7f1897c06640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:36.805 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.815+0000 7f18949a0640 1 -- 172.21.15.114:0/1472722014 <== mon.1 v1:172.21.15.176:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1880001860 con 0x7f1890149dc0 2024-03-29T11:27:36.805 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.815+0000 7f18949a0640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:36.805 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.815+0000 7f1897c06640 10 monclient: get_monmap_and_config success 2024-03-29T11:27:36.805 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.815+0000 7f1897c06640 4 set_mon_vals no callback set 2024-03-29T11:27:36.806 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.815+0000 7f1897c06640 10 monclient: shutdown 2024-03-29T11:27:36.806 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.815+0000 7f1897c06640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:36.806 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.815+0000 7f1897c06640 1 -- 172.21.15.114:0/1472722014 >> v1:172.21.15.176:6789/0 conn(0x7f1890149dc0 legacy=0x7f189014a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:36.806 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.816+0000 7f1897c06640 1 -- 172.21.15.114:0/1472722014 shutdown_connections 2024-03-29T11:27:36.806 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.816+0000 7f1897c06640 1 -- 172.21.15.114:0/1472722014 >> 172.21.15.114:0/1472722014 conn(0x7f18900b35e0 msgr2=0x7f18900b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:36.806 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.817+0000 7f1897c06640 1 -- 172.21.15.114:0/1472722014 shutdown_connections 2024-03-29T11:27:36.806 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.817+0000 7f1897c06640 1 -- 172.21.15.114:0/1472722014 wait complete. 2024-03-29T11:27:36.810 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.818+0000 7f1897c06640 10 monclient: build_initial_monmap 2024-03-29T11:27:36.810 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.818+0000 7f1897c06640 10 monclient: monmap: 2024-03-29T11:27:36.810 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:27:36.810 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:27:36.810 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T11:27:36.810 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T11:27:36.810 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:36.810 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:27:36.810 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:27:36.810 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T11:27:36.810 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T11:27:36.810 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:27:36.811 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.818+0000 7f1897c06640 1 librados: starting msgr at 2024-03-29T11:27:36.811 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.818+0000 7f1897c06640 1 librados: starting objecter 2024-03-29T11:27:36.811 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.818+0000 7f1897c06640 1 Processor -- start 2024-03-29T11:27:36.811 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.819+0000 7f1897c06640 1 -- start start 2024-03-29T11:27:36.811 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.819+0000 7f1897c06640 1 librados: setting wanted keys 2024-03-29T11:27:36.811 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.819+0000 7f1897c06640 1 librados: calling monclient init 2024-03-29T11:27:36.811 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.819+0000 7f1897c06640 10 monclient: init 2024-03-29T11:27:36.811 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.819+0000 7f1897c06640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:36.811 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.819+0000 7f1897c06640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:27:36.811 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.819+0000 7f1897c06640 10 monclient(hunting): picked mon.noname-b con 0x7f18900a9930 addr v1:172.21.15.176:6789/0 2024-03-29T11:27:36.811 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.819+0000 7f1897c06640 10 monclient(hunting): picked mon.noname-a con 0x7f1890149dc0 addr v1:172.21.15.114:6789/0 2024-03-29T11:27:36.811 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.819+0000 7f1897c06640 10 monclient(hunting): picked mon.noname-c con 0x7f189014ff40 addr v1:172.21.15.114:6790/0 2024-03-29T11:27:36.811 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.819+0000 7f1897c06640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1890148860 con 0x7f1890149dc0 2024-03-29T11:27:36.811 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.819+0000 7f1897c06640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18901e4f40 con 0x7f18900a9930 2024-03-29T11:27:36.812 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.819+0000 7f1897c06640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f189013a370 con 0x7f189014ff40 2024-03-29T11:27:36.812 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.819+0000 7f1897c06640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:36.812 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.819+0000 7f1897c06640 10 monclient(hunting): authenticate will time out at 1327.001956s 2024-03-29T11:27:36.819 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.821+0000 7f18959a2640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f18900a9930 0x7f18901e72c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.114:53940/0 (socket says 172.21.15.114:53940) 2024-03-29T11:27:36.819 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.822+0000 7f18959a2640 1 -- 172.21.15.114:0/197010883 learned_addr learned my addr 172.21.15.114:0/197010883 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:27:36.819 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.824+0000 7f18867fc640 10 client.?.objecter ms_handle_connect 0x7f18900a9930 2024-03-29T11:27:36.819 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.824+0000 7f18867fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:36.819 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.824+0000 7f18867fc640 10 client.?.objecter ms_handle_connect 0x7f189014ff40 2024-03-29T11:27:36.819 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.824+0000 7f18867fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:36.819 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.824+0000 7f18867fc640 1 -- 172.21.15.114:0/197010883 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2219217137 0 0) 0x7f18901e4f40 con 0x7f18900a9930 2024-03-29T11:27:36.820 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.824+0000 7f18867fc640 10 monclient(hunting): _negotiate 2024-03-29T11:27:36.821 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.824+0000 7f18867fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:36.821 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.824+0000 7f18867fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:36.821 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.824+0000 7f18867fc640 10 monclient(hunting): my global_id is 4444 2024-03-29T11:27:36.821 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.824+0000 7f18867fc640 1 -- 172.21.15.114:0/197010883 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1874004eb0 con 0x7f18900a9930 2024-03-29T11:27:36.821 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.824+0000 7f18867fc640 10 client.?.objecter ms_handle_connect 0x7f1890149dc0 2024-03-29T11:27:36.821 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.824+0000 7f18867fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:36.821 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.825+0000 7f18867fc640 1 -- 172.21.15.114:0/197010883 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1313782409 0 0) 0x7f1890148860 con 0x7f1890149dc0 2024-03-29T11:27:36.821 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.825+0000 7f18867fc640 10 monclient(hunting): _negotiate 2024-03-29T11:27:36.821 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.825+0000 7f18867fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:36.821 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.825+0000 7f18867fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:36.821 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.825+0000 7f18867fc640 10 monclient(hunting): my global_id is 4473 2024-03-29T11:27:36.821 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.825+0000 7f18867fc640 1 -- 172.21.15.114:0/197010883 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18901e4f40 con 0x7f1890149dc0 2024-03-29T11:27:36.821 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.825+0000 7f18867fc640 1 -- 172.21.15.114:0/197010883 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4251939182 0 0) 0x7f1874004eb0 con 0x7f18900a9930 2024-03-29T11:27:36.822 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.825+0000 7f18867fc640 1 -- 172.21.15.114:0/197010883 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1890148860 con 0x7f18900a9930 2024-03-29T11:27:36.822 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.825+0000 7f18867fc640 1 -- 172.21.15.114:0/197010883 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f1888001ce0 con 0x7f18900a9930 2024-03-29T11:27:36.822 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.825+0000 7f18867fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:36.822 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.825+0000 7f18867fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:27:36.822 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.825+0000 7f18867fc640 10 monclient(hunting): dump: 2024-03-29T11:27:36.822 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:27:36.822 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:27:36.822 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:27:36.822 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:27:36.822 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:36.822 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:27:36.822 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:27:36.822 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:27:36.822 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:27:36.823 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:27:36.823 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.825+0000 7f18867fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:27:36.823 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.825+0000 7f18867fc640 1 -- 172.21.15.114:0/197010883 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2506660496 0 0) 0x7f18901e4f40 con 0x7f1890149dc0 2024-03-29T11:27:36.823 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.825+0000 7f18867fc640 1 -- 172.21.15.114:0/197010883 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1874004eb0 con 0x7f1890149dc0 2024-03-29T11:27:36.823 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.825+0000 7f18867fc640 1 -- 172.21.15.114:0/197010883 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f1880001020 con 0x7f1890149dc0 2024-03-29T11:27:36.823 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.825+0000 7f18867fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:36.823 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.825+0000 7f18867fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:27:36.823 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.825+0000 7f18867fc640 10 monclient(hunting): dump: 2024-03-29T11:27:36.823 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:27:36.823 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:27:36.823 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:27:36.823 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:27:36.824 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:36.824 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:27:36.824 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:27:36.824 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:27:36.824 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:27:36.824 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:27:36.824 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.826+0000 7f18867fc640 1 -- 172.21.15.114:0/197010883 <== mon.0 v1:172.21.15.114:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2436064686 0 0) 0x7f1874004eb0 con 0x7f1890149dc0 2024-03-29T11:27:36.824 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.826+0000 7f18867fc640 1 -- 172.21.15.114:0/197010883 >> v1:172.21.15.114:6790/0 conn(0x7f189014ff40 legacy=0x7f18901e4830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:36.824 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.826+0000 7f18867fc640 1 -- 172.21.15.114:0/197010883 >> v1:172.21.15.176:6789/0 conn(0x7f18900a9930 legacy=0x7f18901e72c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:36.824 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.826+0000 7f18867fc640 10 monclient: _finish_hunting 0 2024-03-29T11:27:36.824 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.826+0000 7f18867fc640 1 monclient: found mon.a 2024-03-29T11:27:36.825 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.826+0000 7f18867fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:36.825 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.826+0000 7f18867fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T11:27:36.825 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.826+0000 7f18867fc640 1 -- 172.21.15.114:0/197010883 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f189013a650 con 0x7f1890149dc0 2024-03-29T11:27:36.825 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.826+0000 7f18867fc640 10 monclient: _finish_auth 0 2024-03-29T11:27:36.825 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.826+0000 7f18867fc640 10 monclient: _check_auth_tickets 2024-03-29T11:27:36.825 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.826+0000 7f18867fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:36.825 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.826+0000 7f1897c06640 5 monclient: authenticate success, global_id 4473 2024-03-29T11:27:36.825 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.826+0000 7f1897c06640 10 monclient: _renew_subs 2024-03-29T11:27:36.825 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.826+0000 7f1897c06640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T11:27:36.826 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.826+0000 7f1897c06640 1 -- 172.21.15.114:0/197010883 --> v1:172.21.15.114:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f189013f580 con 0x7f1890149dc0 2024-03-29T11:27:36.826 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.826+0000 7f1897c06640 10 client.4473.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:27:36.826 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.826+0000 7f1897c06640 10 monclient: _renew_subs 2024-03-29T11:27:36.826 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.826+0000 7f1897c06640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T11:27:36.826 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.826+0000 7f1897c06640 1 -- 172.21.15.114:0/197010883 --> v1:172.21.15.114:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1890140000 con 0x7f1890149dc0 2024-03-29T11:27:36.826 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.826+0000 7f1897c06640 1 librados: init done 2024-03-29T11:27:36.826 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.826+0000 7f18867fc640 1 -- 172.21.15.114:0/197010883 <== mon.0 v1:172.21.15.114:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1880004e00 con 0x7f1890149dc0 2024-03-29T11:27:36.826 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.826+0000 7f18867fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:36.826 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.826+0000 7f18867fc640 1 -- 172.21.15.114:0/197010883 <== mon.0 v1:172.21.15.114:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f1880005190 con 0x7f1890149dc0 2024-03-29T11:27:36.826 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.826+0000 7f18867fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:36.826 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.826+0000 7f18867fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:27:36.827 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.826+0000 7f18867fc640 10 monclient: dump: 2024-03-29T11:27:36.827 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:27:36.827 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:27:36.827 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:27:36.827 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:27:36.827 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:36.827 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:27:36.827 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:27:36.827 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:27:36.827 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:27:36.827 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:27:36.827 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.826+0000 7f1887fff640 4 set_mon_vals no callback set 2024-03-29T11:27:36.827 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.827+0000 7f1897c06640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:36.827 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.827+0000 7f1897c06640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:36.828 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.827+0000 7f1897c06640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T11:27:36.828 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.827+0000 7f1897c06640 1 -- 172.21.15.114:0/197010883 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1878005080 con 0x7f1890149dc0 2024-03-29T11:27:36.828 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.829+0000 7f18867fc640 1 -- 172.21.15.114:0/197010883 <== mon.0 v1:172.21.15.114:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1594292461 0 0) 0x7f1880001ad0 con 0x7f1890149dc0 2024-03-29T11:27:36.828 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.829+0000 7f18867fc640 1 -- 172.21.15.114:0/197010883 <== mon.0 v1:172.21.15.114:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3238987313 0 0) 0x7f1880084c20 con 0x7f1890149dc0 2024-03-29T11:27:36.828 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.829+0000 7f18867fc640 10 client.4473.objecter ms_dispatch 0x7f1890000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:27:36.828 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.829+0000 7f18867fc640 3 client.4473.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:27:36.828 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.829+0000 7f18867fc640 3 client.4473.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:27:36.828 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.829+0000 7f18867fc640 20 client.4473.objecter dump_active .. 0 homeless 2024-03-29T11:27:36.828 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.830+0000 7f18959a2640 10 monclient: get_auth_request con 0x7f187406df10 auth_method 0 2024-03-29T11:27:36.830 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.830+0000 7f18867fc640 10 client.4473.objecter ms_handle_connect 0x7f187406df10 2024-03-29T11:27:36.830 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.839+0000 7f18867fc640 1 -- 172.21.15.114:0/197010883 <== mon.0 v1:172.21.15.114:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1880050500 con 0x7f1890149dc0 2024-03-29T11:27:36.831 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.839+0000 7f18867fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:36.831 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:36.839+0000 7f18867fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:27:37.162 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:37.173+0000 7f1897c06640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:37.162 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:37.173+0000 7f1897c06640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:37.162 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:37.173+0000 7f1897c06640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T11:27:37.162 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:37.173+0000 7f1897c06640 1 -- 172.21.15.114:0/197010883 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1878005330 con 0x7f1890149dc0 2024-03-29T11:27:37.164 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:37.175+0000 7f18867fc640 1 -- 172.21.15.114:0/197010883 <== mon.0 v1:172.21.15.114:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2858224613) 0x7f1880052430 con 0x7f1890149dc0 2024-03-29T11:27:37.164 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:37.176+0000 7f18867fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:37.164 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:37.176+0000 7f18867fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:27:37.164 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T11:27:37.165 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"b0de2b9b-c45a-44c7-b6ee-e83b39a1e108","created":"2024-03-29T11:20:49.420660+0000","modified":"2024-03-29T11:20:51.700167+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:51.700167+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a64785d3-6b11-499a-b706-cc827ab96a74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5882dadb-e7a8-4604-ba9a-418013c106c8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d7e8c409-bc66-4e4a-bbdb-420bad31388e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0682989-3595-4cd6-90a5-ab9111c278e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"67068120-e3ff-4817-bd06-44feb7479a35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09e8b-8a85-4997-bdea-75d754e6ee67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2f0d9ead-f47f-4cb3-92a4-0a5ad23cd831","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fdb6a25f-6bd8-4c65-9b9b-b460ba7673a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:27:37.167 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:37.179+0000 7f1897c06640 10 librados: watch_flush enter 2024-03-29T11:27:37.168 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:37.179+0000 7f1897c06640 10 librados: watch_flush exit 2024-03-29T11:27:37.168 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:37.179+0000 7f1897c06640 20 client.4473.objecter shutdown clearing up homeless session... 2024-03-29T11:27:37.168 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:37.179+0000 7f1897c06640 10 client.4473.objecter successfully canceled tick 2024-03-29T11:27:37.168 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:37.179+0000 7f1897c06640 1 -- 172.21.15.114:0/197010883 >> v1:172.21.15.114:6816/4038755055 conn(0x7f187406df10 legacy=0x7f18740703d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:37.168 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:37.179+0000 7f1897c06640 10 monclient: shutdown 2024-03-29T11:27:37.168 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:37.179+0000 7f1897c06640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:37.168 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:37.179+0000 7f1897c06640 1 -- 172.21.15.114:0/197010883 >> v1:172.21.15.114:6789/0 conn(0x7f1890149dc0 legacy=0x7f18901e4120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:37.169 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:37.180+0000 7f1897c06640 1 -- 172.21.15.114:0/197010883 shutdown_connections 2024-03-29T11:27:37.169 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:37.180+0000 7f1897c06640 1 -- 172.21.15.114:0/197010883 >> 172.21.15.114:0/197010883 conn(0x7f18900b35e0 msgr2=0x7f1890141dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:37.169 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:37.180+0000 7f1897c06640 1 -- 172.21.15.114:0/197010883 shutdown_connections 2024-03-29T11:27:37.169 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:37.180+0000 7f1897c06640 1 -- 172.21.15.114:0/197010883 wait complete. 2024-03-29T11:27:37.169 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:37.180+0000 7f1897c06640 1 librados: shutdown 2024-03-29T11:27:37.188 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:27:43.995 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:27:44.142 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.153+0000 7fe96f06e640 10 monclient: get_monmap_and_config 2024-03-29T11:27:44.142 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.153+0000 7fe96f06e640 10 monclient: build_initial_monmap 2024-03-29T11:27:44.142 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.153+0000 7fe96f06e640 10 monclient: monmap: 2024-03-29T11:27:44.142 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:27:44.142 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:27:44.142 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:27:44.154306+0000 2024-03-29T11:27:44.142 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:27:44.154306+0000 2024-03-29T11:27:44.142 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:44.142 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:27:44.142 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:27:44.143 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T11:27:44.143 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T11:27:44.143 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:27:44.143 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.154+0000 7fe96f06e640 1 Processor -- start 2024-03-29T11:27:44.143 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.154+0000 7fe96f06e640 1 -- start start 2024-03-29T11:27:44.144 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.154+0000 7fe96f06e640 10 monclient: init 2024-03-29T11:27:44.144 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.155+0000 7fe96f06e640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:44.144 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.155+0000 7fe96f06e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:27:44.145 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.155+0000 7fe96f06e640 10 monclient(hunting): picked mon.noname-a con 0x7fe968071f40 addr v1:172.21.15.114:6789/0 2024-03-29T11:27:44.145 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.156+0000 7fe96f06e640 10 monclient(hunting): picked mon.noname-b con 0x7fe9680adcf0 addr v1:172.21.15.114:6790/0 2024-03-29T11:27:44.145 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.156+0000 7fe96f06e640 10 monclient(hunting): picked mon.noname-c con 0x7fe96813f6e0 addr v1:172.21.15.176:6789/0 2024-03-29T11:27:44.145 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.156+0000 7fe96f06e640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9680aef80 con 0x7fe968071f40 2024-03-29T11:27:44.145 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.156+0000 7fe96f06e640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9680affc0 con 0x7fe96813f6e0 2024-03-29T11:27:44.146 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.156+0000 7fe96f06e640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9680aad20 con 0x7fe9680adcf0 2024-03-29T11:27:44.146 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.156+0000 7fe96f06e640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:44.146 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.156+0000 7fe96f06e640 10 monclient(hunting): authenticate will time out at 1334.338838s 2024-03-29T11:27:44.146 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.158+0000 7fe96e06c640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fe968071f40 0x7fe96813efd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:44635/0 (socket says 172.21.15.114:44635) 2024-03-29T11:27:44.147 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.159+0000 7fe96d86b640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7fe9680adcf0 0x7fe968142040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:35400/0 (socket says 172.21.15.114:35400) 2024-03-29T11:27:44.147 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.159+0000 7fe96d86b640 1 -- 172.21.15.114:0/1044075764 learned_addr learned my addr 172.21.15.114:0/1044075764 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:27:44.151 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.159+0000 7fe96d06a640 1 -- 172.21.15.114:0/1044075764 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1469207483 0 0) 0x7fe9680aad20 con 0x7fe9680adcf0 2024-03-29T11:27:44.151 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.159+0000 7fe96d06a640 10 monclient(hunting): _negotiate 2024-03-29T11:27:44.151 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.159+0000 7fe96d06a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:44.151 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.159+0000 7fe96d06a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:44.151 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.159+0000 7fe96d06a640 10 monclient(hunting): my global_id is 4490 2024-03-29T11:27:44.152 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.159+0000 7fe96d06a640 1 -- 172.21.15.114:0/1044075764 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe954003520 con 0x7fe9680adcf0 2024-03-29T11:27:44.152 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.159+0000 7fe96d06a640 1 -- 172.21.15.114:0/1044075764 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1307563131 0 0) 0x7fe954003520 con 0x7fe9680adcf0 2024-03-29T11:27:44.152 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.159+0000 7fe96d06a640 1 -- 172.21.15.114:0/1044075764 >> v1:172.21.15.176:6789/0 conn(0x7fe96813f6e0 legacy=0x7fe968142750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:27:44.152 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.159+0000 7fe96d06a640 1 -- 172.21.15.114:0/1044075764 >> v1:172.21.15.114:6789/0 conn(0x7fe968071f40 legacy=0x7fe96813efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:27:44.152 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.159+0000 7fe96d06a640 10 monclient: _finish_hunting 0 2024-03-29T11:27:44.152 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.159+0000 7fe96d06a640 1 monclient: found mon.noname-b 2024-03-29T11:27:44.152 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.159+0000 7fe96d06a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:44.152 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.159+0000 7fe96d06a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.114:6790/0 2024-03-29T11:27:44.152 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.159+0000 7fe96d06a640 1 -- 172.21.15.114:0/1044075764 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9680ab000 con 0x7fe9680adcf0 2024-03-29T11:27:44.152 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.159+0000 7fe96d06a640 10 monclient: _finish_auth 0 2024-03-29T11:27:44.152 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.159+0000 7fe96d06a640 10 monclient: _check_auth_tickets 2024-03-29T11:27:44.152 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.159+0000 7fe96d06a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:44.152 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.159+0000 7fe96d06a640 1 -- 172.21.15.114:0/1044075764 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fe964001690 con 0x7fe9680adcf0 2024-03-29T11:27:44.152 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.159+0000 7fe96d06a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:44.153 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.159+0000 7fe96d06a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:27:44.153 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.159+0000 7fe96d06a640 10 monclient: dump: 2024-03-29T11:27:44.153 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:27:44.153 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:27:44.153 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:27:44.153 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:27:44.153 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:44.153 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:27:44.153 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:27:44.153 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:27:44.153 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:27:44.153 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:27:44.153 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.159+0000 7fe96f06e640 5 monclient: authenticate success, global_id 4490 2024-03-29T11:27:44.153 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.159+0000 7fe96f06e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:44.153 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.160+0000 7fe96d06a640 1 -- 172.21.15.114:0/1044075764 <== mon.2 v1:172.21.15.114:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe9640018b0 con 0x7fe9680adcf0 2024-03-29T11:27:44.154 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.160+0000 7fe96d06a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:44.154 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.160+0000 7fe96d06a640 1 -- 172.21.15.114:0/1044075764 <== mon.2 v1:172.21.15.114:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fe964001c40 con 0x7fe9680adcf0 2024-03-29T11:27:44.154 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.160+0000 7fe96f06e640 10 monclient: get_monmap_and_config success 2024-03-29T11:27:44.154 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.160+0000 7fe96f06e640 4 set_mon_vals no callback set 2024-03-29T11:27:44.154 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.160+0000 7fe96d06a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:44.154 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.160+0000 7fe96d06a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:44.154 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.160+0000 7fe96d06a640 10 monclient: dump: 2024-03-29T11:27:44.154 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:27:44.154 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:27:44.154 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:27:44.154 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:27:44.154 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:44.154 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:27:44.154 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:27:44.154 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:27:44.155 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:27:44.155 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:27:44.155 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.161+0000 7fe96f06e640 10 monclient: shutdown 2024-03-29T11:27:44.155 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.161+0000 7fe96f06e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:44.155 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.161+0000 7fe96f06e640 1 -- 172.21.15.114:0/1044075764 >> v1:172.21.15.114:6790/0 conn(0x7fe9680adcf0 legacy=0x7fe968142040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:44.155 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.161+0000 7fe96f06e640 1 -- 172.21.15.114:0/1044075764 shutdown_connections 2024-03-29T11:27:44.155 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.161+0000 7fe96f06e640 1 -- 172.21.15.114:0/1044075764 >> 172.21.15.114:0/1044075764 conn(0x7fe9680673e0 msgr2=0x7fe9680b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:44.156 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.164+0000 7fe96f06e640 1 -- 172.21.15.114:0/1044075764 shutdown_connections 2024-03-29T11:27:44.156 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.164+0000 7fe96f06e640 1 -- 172.21.15.114:0/1044075764 wait complete. 2024-03-29T11:27:44.156 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.165+0000 7fe96f06e640 10 monclient: build_initial_monmap 2024-03-29T11:27:44.156 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.165+0000 7fe96f06e640 10 monclient: monmap: 2024-03-29T11:27:44.156 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:27:44.156 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:27:44.156 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T11:27:44.156 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T11:27:44.156 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:44.156 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:27:44.157 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:27:44.157 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T11:27:44.157 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T11:27:44.157 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:27:44.157 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.166+0000 7fe96f06e640 1 librados: starting msgr at 2024-03-29T11:27:44.157 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.167+0000 7fe96f06e640 1 librados: starting objecter 2024-03-29T11:27:44.157 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.167+0000 7fe96f06e640 1 Processor -- start 2024-03-29T11:27:44.157 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.168+0000 7fe96f06e640 1 -- start start 2024-03-29T11:27:44.157 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.168+0000 7fe96f06e640 1 librados: setting wanted keys 2024-03-29T11:27:44.158 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.168+0000 7fe96f06e640 1 librados: calling monclient init 2024-03-29T11:27:44.158 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.168+0000 7fe96f06e640 10 monclient: init 2024-03-29T11:27:44.158 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.168+0000 7fe96f06e640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:44.158 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.168+0000 7fe96f06e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:27:44.158 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.169+0000 7fe96f06e640 10 monclient(hunting): picked mon.noname-b con 0x7fe968071f40 addr v1:172.21.15.176:6789/0 2024-03-29T11:27:44.158 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.169+0000 7fe96f06e640 10 monclient(hunting): picked mon.noname-a con 0x7fe9680adcf0 addr v1:172.21.15.114:6789/0 2024-03-29T11:27:44.158 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.169+0000 7fe96f06e640 10 monclient(hunting): picked mon.noname-c con 0x7fe96813f6e0 addr v1:172.21.15.114:6790/0 2024-03-29T11:27:44.158 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.169+0000 7fe96f06e640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9681d4a40 con 0x7fe9680adcf0 2024-03-29T11:27:44.158 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.169+0000 7fe96f06e640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe96814dd40 con 0x7fe968071f40 2024-03-29T11:27:44.158 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.169+0000 7fe96f06e640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe96814ded0 con 0x7fe96813f6e0 2024-03-29T11:27:44.158 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.169+0000 7fe96f06e640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:44.158 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.169+0000 7fe96f06e640 10 monclient(hunting): authenticate will time out at 1334.351176s 2024-03-29T11:27:44.161 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.172+0000 7fe96d86b640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fe9680adcf0 0x7fe9681da410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:45093/0 (socket says 172.21.15.114:45093) 2024-03-29T11:27:44.161 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.172+0000 7fe96e86d640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7fe96813f6e0 0x7fe9681338d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:37718/0 (socket says 172.21.15.114:37718) 2024-03-29T11:27:44.161 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.172+0000 7fe96e86d640 1 -- 172.21.15.114:0/639177368 learned_addr learned my addr 172.21.15.114:0/639177368 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:27:44.162 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.173+0000 7fe95effd640 10 client.?.objecter ms_handle_connect 0x7fe96813f6e0 2024-03-29T11:27:44.163 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.173+0000 7fe95effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:44.163 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.173+0000 7fe95effd640 10 client.?.objecter ms_handle_connect 0x7fe9680adcf0 2024-03-29T11:27:44.163 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.173+0000 7fe95effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:44.163 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.173+0000 7fe95effd640 1 -- 172.21.15.114:0/639177368 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2746376712 0 0) 0x7fe96814ded0 con 0x7fe96813f6e0 2024-03-29T11:27:44.163 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.173+0000 7fe95effd640 10 monclient(hunting): _negotiate 2024-03-29T11:27:44.163 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.173+0000 7fe95effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:44.163 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.173+0000 7fe95effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:44.163 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.173+0000 7fe95effd640 10 monclient(hunting): my global_id is 4493 2024-03-29T11:27:44.163 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.173+0000 7fe95effd640 1 -- 172.21.15.114:0/639177368 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe94c004eb0 con 0x7fe96813f6e0 2024-03-29T11:27:44.163 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.173+0000 7fe95effd640 1 -- 172.21.15.114:0/639177368 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2716863623 0 0) 0x7fe9681d4a40 con 0x7fe9680adcf0 2024-03-29T11:27:44.163 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.173+0000 7fe95effd640 10 monclient(hunting): _negotiate 2024-03-29T11:27:44.163 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.173+0000 7fe95effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:44.163 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.174+0000 7fe95effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:44.163 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.174+0000 7fe95effd640 10 monclient(hunting): my global_id is 4476 2024-03-29T11:27:44.164 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.174+0000 7fe95effd640 1 -- 172.21.15.114:0/639177368 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe96814ded0 con 0x7fe9680adcf0 2024-03-29T11:27:44.164 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.174+0000 7fe95effd640 10 client.?.objecter ms_handle_connect 0x7fe968071f40 2024-03-29T11:27:44.164 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.174+0000 7fe95effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:44.168 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.174+0000 7fe95effd640 1 -- 172.21.15.114:0/639177368 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3068359419 0 0) 0x7fe94c004eb0 con 0x7fe96813f6e0 2024-03-29T11:27:44.168 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.174+0000 7fe95effd640 1 -- 172.21.15.114:0/639177368 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe9681d4a40 con 0x7fe96813f6e0 2024-03-29T11:27:44.168 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.174+0000 7fe95effd640 1 -- 172.21.15.114:0/639177368 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fe960001ad0 con 0x7fe96813f6e0 2024-03-29T11:27:44.168 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.174+0000 7fe95effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:44.168 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.174+0000 7fe95effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:27:44.168 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.174+0000 7fe95effd640 10 monclient(hunting): dump: 2024-03-29T11:27:44.169 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:27:44.169 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:27:44.169 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:27:44.169 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:27:44.169 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:44.169 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:27:44.169 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:27:44.169 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:27:44.169 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:27:44.169 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:27:44.169 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.174+0000 7fe95effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:27:44.169 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.174+0000 7fe95effd640 1 -- 172.21.15.114:0/639177368 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1837790051 0 0) 0x7fe96814dd40 con 0x7fe968071f40 2024-03-29T11:27:44.169 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.174+0000 7fe95effd640 10 monclient(hunting): _negotiate 2024-03-29T11:27:44.169 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.174+0000 7fe95effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:44.169 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.174+0000 7fe95effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:44.170 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.174+0000 7fe95effd640 10 monclient(hunting): my global_id is 4447 2024-03-29T11:27:44.170 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.174+0000 7fe95effd640 1 -- 172.21.15.114:0/639177368 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe94c004eb0 con 0x7fe968071f40 2024-03-29T11:27:44.170 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.174+0000 7fe95effd640 1 -- 172.21.15.114:0/639177368 <== mon.2 v1:172.21.15.114:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3870585439 0 0) 0x7fe9681d4a40 con 0x7fe96813f6e0 2024-03-29T11:27:44.170 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.174+0000 7fe95effd640 1 -- 172.21.15.114:0/639177368 >> v1:172.21.15.176:6789/0 conn(0x7fe968071f40 legacy=0x7fe9681d3d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:44.170 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.174+0000 7fe95effd640 1 -- 172.21.15.114:0/639177368 >> v1:172.21.15.114:6789/0 conn(0x7fe9680adcf0 legacy=0x7fe9681da410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:44.170 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.174+0000 7fe95effd640 10 monclient: _finish_hunting 0 2024-03-29T11:27:44.170 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.174+0000 7fe95effd640 1 monclient: found mon.c 2024-03-29T11:27:44.170 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.174+0000 7fe95effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:44.170 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.174+0000 7fe95effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:27:44.170 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.174+0000 7fe95effd640 1 -- 172.21.15.114:0/639177368 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9681dab20 con 0x7fe96813f6e0 2024-03-29T11:27:44.171 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.174+0000 7fe95effd640 10 monclient: _finish_auth 0 2024-03-29T11:27:44.171 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.174+0000 7fe95effd640 10 monclient: _check_auth_tickets 2024-03-29T11:27:44.171 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.174+0000 7fe95effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:44.171 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.174+0000 7fe96f06e640 5 monclient: authenticate success, global_id 4493 2024-03-29T11:27:44.171 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.174+0000 7fe96f06e640 10 monclient: _renew_subs 2024-03-29T11:27:44.171 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.174+0000 7fe96f06e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:27:44.171 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.174+0000 7fe96f06e640 1 -- 172.21.15.114:0/639177368 --> v1:172.21.15.114:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9681dad70 con 0x7fe96813f6e0 2024-03-29T11:27:44.171 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.174+0000 7fe96f06e640 10 client.4493.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:27:44.171 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.174+0000 7fe96f06e640 10 monclient: _renew_subs 2024-03-29T11:27:44.171 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.174+0000 7fe96f06e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:27:44.171 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.174+0000 7fe96f06e640 1 -- 172.21.15.114:0/639177368 --> v1:172.21.15.114:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe968140290 con 0x7fe96813f6e0 2024-03-29T11:27:44.171 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.174+0000 7fe96f06e640 1 librados: init done 2024-03-29T11:27:44.172 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.175+0000 7fe95effd640 1 -- 172.21.15.114:0/639177368 <== mon.2 v1:172.21.15.114:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe960001c30 con 0x7fe96813f6e0 2024-03-29T11:27:44.172 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.175+0000 7fe95effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:44.172 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.175+0000 7fe95effd640 1 -- 172.21.15.114:0/639177368 <== mon.2 v1:172.21.15.114:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fe960004d80 con 0x7fe96813f6e0 2024-03-29T11:27:44.172 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.175+0000 7fe95effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:44.172 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.175+0000 7fe95effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:44.172 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.175+0000 7fe95effd640 10 monclient: dump: 2024-03-29T11:27:44.172 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:27:44.172 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:27:44.172 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:27:44.172 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:27:44.172 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:44.172 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:27:44.172 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:27:44.172 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:27:44.172 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:27:44.173 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:27:44.173 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.175+0000 7fe96c869640 4 set_mon_vals no callback set 2024-03-29T11:27:44.173 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.177+0000 7fe95effd640 1 -- 172.21.15.114:0/639177368 <== mon.2 v1:172.21.15.114:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1594292461 0 0) 0x7fe96001ad50 con 0x7fe96813f6e0 2024-03-29T11:27:44.173 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.177+0000 7fe95effd640 1 -- 172.21.15.114:0/639177368 <== mon.2 v1:172.21.15.114:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3238987313 0 0) 0x7fe960084cb0 con 0x7fe96813f6e0 2024-03-29T11:27:44.173 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.177+0000 7fe95effd640 10 client.4493.objecter ms_dispatch 0x7fe968000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:27:44.173 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.177+0000 7fe95effd640 3 client.4493.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:27:44.173 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.177+0000 7fe95effd640 3 client.4493.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:27:44.173 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.177+0000 7fe95effd640 20 client.4493.objecter dump_active .. 0 homeless 2024-03-29T11:27:44.173 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.177+0000 7fe96e06c640 10 monclient: get_auth_request con 0x7fe94c06df70 auth_method 0 2024-03-29T11:27:44.173 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.178+0000 7fe95effd640 10 client.4493.objecter ms_handle_connect 0x7fe94c06df70 2024-03-29T11:27:44.173 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.181+0000 7fe96f06e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:44.174 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.181+0000 7fe96f06e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:44.174 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.181+0000 7fe96f06e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:27:44.174 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.181+0000 7fe96f06e640 1 -- 172.21.15.114:0/639177368 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe950005080 con 0x7fe96813f6e0 2024-03-29T11:27:44.179 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.191+0000 7fe95effd640 1 -- 172.21.15.114:0/639177368 <== mon.2 v1:172.21.15.114:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe960050590 con 0x7fe96813f6e0 2024-03-29T11:27:44.180 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.191+0000 7fe95effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:44.180 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.191+0000 7fe95effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:27:44.511 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.522+0000 7fe96f06e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:44.511 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.522+0000 7fe96f06e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:44.511 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.522+0000 7fe96f06e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:27:44.511 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.522+0000 7fe96f06e640 1 -- 172.21.15.114:0/639177368 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe950005330 con 0x7fe96813f6e0 2024-03-29T11:27:44.514 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.525+0000 7fe95effd640 1 -- 172.21.15.114:0/639177368 <== mon.2 v1:172.21.15.114:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2858224613) 0x7fe9600524c0 con 0x7fe96813f6e0 2024-03-29T11:27:44.514 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.526+0000 7fe95effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:44.514 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.526+0000 7fe95effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:27:44.516 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T11:27:44.516 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"b0de2b9b-c45a-44c7-b6ee-e83b39a1e108","created":"2024-03-29T11:20:49.420660+0000","modified":"2024-03-29T11:20:51.700167+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:51.700167+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a64785d3-6b11-499a-b706-cc827ab96a74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5882dadb-e7a8-4604-ba9a-418013c106c8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d7e8c409-bc66-4e4a-bbdb-420bad31388e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0682989-3595-4cd6-90a5-ab9111c278e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"67068120-e3ff-4817-bd06-44feb7479a35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09e8b-8a85-4997-bdea-75d754e6ee67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2f0d9ead-f47f-4cb3-92a4-0a5ad23cd831","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fdb6a25f-6bd8-4c65-9b9b-b460ba7673a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:27:44.519 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.530+0000 7fe96f06e640 10 librados: watch_flush enter 2024-03-29T11:27:44.519 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.530+0000 7fe96f06e640 10 librados: watch_flush exit 2024-03-29T11:27:44.519 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.530+0000 7fe96f06e640 20 client.4493.objecter shutdown clearing up homeless session... 2024-03-29T11:27:44.519 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.530+0000 7fe96f06e640 10 client.4493.objecter successfully canceled tick 2024-03-29T11:27:44.519 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.530+0000 7fe96f06e640 1 -- 172.21.15.114:0/639177368 >> v1:172.21.15.114:6816/4038755055 conn(0x7fe94c06df70 legacy=0x7fe94c070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:44.519 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.530+0000 7fe96f06e640 10 monclient: shutdown 2024-03-29T11:27:44.519 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.530+0000 7fe96f06e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:44.519 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.530+0000 7fe96f06e640 1 -- 172.21.15.114:0/639177368 >> v1:172.21.15.114:6790/0 conn(0x7fe96813f6e0 legacy=0x7fe9681338d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:44.519 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.530+0000 7fe96f06e640 1 -- 172.21.15.114:0/639177368 shutdown_connections 2024-03-29T11:27:44.519 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.530+0000 7fe96f06e640 1 -- 172.21.15.114:0/639177368 >> 172.21.15.114:0/639177368 conn(0x7fe9680673e0 msgr2=0x7fe968069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:44.519 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.530+0000 7fe96f06e640 1 -- 172.21.15.114:0/639177368 shutdown_connections 2024-03-29T11:27:44.519 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.530+0000 7fe96f06e640 1 -- 172.21.15.114:0/639177368 wait complete. 2024-03-29T11:27:44.519 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:44.530+0000 7fe96f06e640 1 librados: shutdown 2024-03-29T11:27:44.535 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:27:51.342 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:27:51.494 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.504+0000 7ff4ce952640 10 monclient: get_monmap_and_config 2024-03-29T11:27:51.494 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.504+0000 7ff4ce952640 10 monclient: build_initial_monmap 2024-03-29T11:27:51.494 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.504+0000 7ff4ce952640 10 monclient: monmap: 2024-03-29T11:27:51.494 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:27:51.494 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:27:51.494 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:27:51.505884+0000 2024-03-29T11:27:51.494 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:27:51.505884+0000 2024-03-29T11:27:51.494 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:51.494 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:27:51.494 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:27:51.494 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T11:27:51.494 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T11:27:51.495 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:27:51.495 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.506+0000 7ff4ce952640 1 Processor -- start 2024-03-29T11:27:51.495 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.506+0000 7ff4ce952640 1 -- start start 2024-03-29T11:27:51.495 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.506+0000 7ff4ce952640 10 monclient: init 2024-03-29T11:27:51.495 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.506+0000 7ff4ce952640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:51.495 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.506+0000 7ff4ce952640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:27:51.496 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.507+0000 7ff4ce952640 10 monclient(hunting): picked mon.noname-c con 0x7ff4c806d9d0 addr v1:172.21.15.176:6789/0 2024-03-29T11:27:51.496 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.507+0000 7ff4ce952640 10 monclient(hunting): picked mon.noname-a con 0x7ff4c813ef30 addr v1:172.21.15.114:6789/0 2024-03-29T11:27:51.496 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.507+0000 7ff4ce952640 10 monclient(hunting): picked mon.noname-b con 0x7ff4c80b0860 addr v1:172.21.15.114:6790/0 2024-03-29T11:27:51.496 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.507+0000 7ff4ce952640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4c80aed10 con 0x7ff4c813ef30 2024-03-29T11:27:51.496 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.507+0000 7ff4ce952640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4c80ab1b0 con 0x7ff4c806d9d0 2024-03-29T11:27:51.496 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.507+0000 7ff4ce952640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4c80ab490 con 0x7ff4c80b0860 2024-03-29T11:27:51.496 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.507+0000 7ff4ce952640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:51.496 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.507+0000 7ff4ce952640 10 monclient(hunting): authenticate will time out at 1341.689224s 2024-03-29T11:27:51.497 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.508+0000 7ff4cceef640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7ff4c80b0860 0x7ff4c8142a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:33034/0 (socket says 172.21.15.114:33034) 2024-03-29T11:27:51.498 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.508+0000 7ff4cceef640 1 -- 172.21.15.114:0/2953427135 learned_addr learned my addr 172.21.15.114:0/2953427135 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:27:51.498 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.508+0000 7ff4c6ffd640 1 -- 172.21.15.114:0/2953427135 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2266460407 0 0) 0x7ff4c80ab490 con 0x7ff4c80b0860 2024-03-29T11:27:51.498 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.508+0000 7ff4c6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:27:51.498 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.508+0000 7ff4c6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:51.498 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.508+0000 7ff4c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:51.498 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.508+0000 7ff4c6ffd640 10 monclient(hunting): my global_id is 4496 2024-03-29T11:27:51.498 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.508+0000 7ff4c6ffd640 1 -- 172.21.15.114:0/2953427135 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4b0003520 con 0x7ff4c80b0860 2024-03-29T11:27:51.499 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.510+0000 7ff4c6ffd640 1 -- 172.21.15.114:0/2953427135 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2068139941 0 0) 0x7ff4b0003520 con 0x7ff4c80b0860 2024-03-29T11:27:51.499 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.510+0000 7ff4c6ffd640 1 -- 172.21.15.114:0/2953427135 >> v1:172.21.15.176:6789/0 conn(0x7ff4c806d9d0 legacy=0x7ff4c813e820 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:27:51.502 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.513+0000 7ff4c6ffd640 1 -- 172.21.15.114:0/2953427135 >> v1:172.21.15.114:6789/0 conn(0x7ff4c813ef30 legacy=0x7ff4c80af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:51.502 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.513+0000 7ff4c6ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:27:51.503 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.513+0000 7ff4c6ffd640 1 monclient: found mon.noname-b 2024-03-29T11:27:51.503 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.513+0000 7ff4c6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:51.503 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.513+0000 7ff4c6ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.114:6790/0 2024-03-29T11:27:51.503 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.513+0000 7ff4c6ffd640 1 -- 172.21.15.114:0/2953427135 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4c80ab770 con 0x7ff4c80b0860 2024-03-29T11:27:51.503 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.513+0000 7ff4c6ffd640 10 monclient: _finish_auth 0 2024-03-29T11:27:51.503 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.513+0000 7ff4c6ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:27:51.503 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.513+0000 7ff4c6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:51.503 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.513+0000 7ff4ce952640 5 monclient: authenticate success, global_id 4496 2024-03-29T11:27:51.503 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.513+0000 7ff4ce952640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:51.503 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.513+0000 7ff4c6ffd640 1 -- 172.21.15.114:0/2953427135 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7ff4b8001690 con 0x7ff4c80b0860 2024-03-29T11:27:51.503 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.513+0000 7ff4c6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:51.503 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.513+0000 7ff4c6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:27:51.503 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.513+0000 7ff4c6ffd640 10 monclient: dump: 2024-03-29T11:27:51.503 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:27:51.504 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:27:51.504 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:27:51.504 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:27:51.504 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:51.504 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:27:51.504 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:27:51.504 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:27:51.504 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:27:51.504 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:27:51.504 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.513+0000 7ff4ce952640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:51.504 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.513+0000 7ff4c6ffd640 1 -- 172.21.15.114:0/2953427135 <== mon.2 v1:172.21.15.114:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff4b8001860 con 0x7ff4c80b0860 2024-03-29T11:27:51.504 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.513+0000 7ff4c6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:51.504 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.513+0000 7ff4ce952640 10 monclient: get_monmap_and_config success 2024-03-29T11:27:51.505 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.513+0000 7ff4ce952640 4 set_mon_vals no callback set 2024-03-29T11:27:51.505 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.513+0000 7ff4c6ffd640 1 -- 172.21.15.114:0/2953427135 <== mon.2 v1:172.21.15.114:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7ff4b8001c70 con 0x7ff4c80b0860 2024-03-29T11:27:51.505 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.513+0000 7ff4c6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:51.505 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.513+0000 7ff4c6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:51.505 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.513+0000 7ff4c6ffd640 10 monclient: dump: 2024-03-29T11:27:51.505 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:27:51.505 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:27:51.505 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:27:51.505 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:27:51.505 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:51.505 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:27:51.505 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:27:51.505 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:27:51.506 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:27:51.506 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:27:51.507 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.514+0000 7ff4ce952640 10 monclient: shutdown 2024-03-29T11:27:51.507 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.514+0000 7ff4ce952640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:51.507 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.514+0000 7ff4ce952640 1 -- 172.21.15.114:0/2953427135 >> v1:172.21.15.114:6790/0 conn(0x7ff4c80b0860 legacy=0x7ff4c8142a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:51.507 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.514+0000 7ff4ce952640 1 -- 172.21.15.114:0/2953427135 shutdown_connections 2024-03-29T11:27:51.507 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.514+0000 7ff4ce952640 1 -- 172.21.15.114:0/2953427135 >> 172.21.15.114:0/2953427135 conn(0x7ff4c806b660 msgr2=0x7ff4c806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:51.507 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.516+0000 7ff4ce952640 1 -- 172.21.15.114:0/2953427135 shutdown_connections 2024-03-29T11:27:51.507 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.516+0000 7ff4ce952640 1 -- 172.21.15.114:0/2953427135 wait complete. 2024-03-29T11:27:51.507 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.517+0000 7ff4ce952640 10 monclient: build_initial_monmap 2024-03-29T11:27:51.507 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.517+0000 7ff4ce952640 10 monclient: monmap: 2024-03-29T11:27:51.507 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:27:51.508 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:27:51.508 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T11:27:51.508 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T11:27:51.508 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:51.508 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:27:51.508 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:27:51.508 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T11:27:51.508 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T11:27:51.508 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:27:51.508 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.518+0000 7ff4ce952640 1 librados: starting msgr at 2024-03-29T11:27:51.508 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.518+0000 7ff4ce952640 1 librados: starting objecter 2024-03-29T11:27:51.508 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.518+0000 7ff4ce952640 1 Processor -- start 2024-03-29T11:27:51.512 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.518+0000 7ff4ce952640 1 -- start start 2024-03-29T11:27:51.512 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.518+0000 7ff4ce952640 1 librados: setting wanted keys 2024-03-29T11:27:51.512 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.518+0000 7ff4ce952640 1 librados: calling monclient init 2024-03-29T11:27:51.512 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.518+0000 7ff4ce952640 10 monclient: init 2024-03-29T11:27:51.512 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.519+0000 7ff4ce952640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:51.512 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.519+0000 7ff4ce952640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:27:51.512 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.519+0000 7ff4ce952640 10 monclient(hunting): picked mon.noname-b con 0x7ff4c806d9d0 addr v1:172.21.15.176:6789/0 2024-03-29T11:27:51.512 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.519+0000 7ff4ce952640 10 monclient(hunting): picked mon.noname-c con 0x7ff4c80b0860 addr v1:172.21.15.114:6790/0 2024-03-29T11:27:51.512 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.519+0000 7ff4ce952640 10 monclient(hunting): picked mon.noname-a con 0x7ff4c813ef30 addr v1:172.21.15.114:6789/0 2024-03-29T11:27:51.512 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.519+0000 7ff4ce952640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4c80b1a90 con 0x7ff4c813ef30 2024-03-29T11:27:51.512 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.519+0000 7ff4ce952640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4c80b3bc0 con 0x7ff4c806d9d0 2024-03-29T11:27:51.513 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.519+0000 7ff4ce952640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4c80b2a10 con 0x7ff4c80b0860 2024-03-29T11:27:51.513 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.519+0000 7ff4ce952640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:51.513 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.519+0000 7ff4ce952640 10 monclient(hunting): authenticate will time out at 1341.701595s 2024-03-29T11:27:51.513 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.519+0000 7ff4c77fe640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7ff4c80b0860 0x7ff4c81362f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:33770/0 (socket says 172.21.15.114:33770) 2024-03-29T11:27:51.513 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.519+0000 7ff4c77fe640 1 -- 172.21.15.114:0/784208792 learned_addr learned my addr 172.21.15.114:0/784208792 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:27:51.513 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.519+0000 7ff4c4ff9640 10 client.?.objecter ms_handle_connect 0x7ff4c80b0860 2024-03-29T11:27:51.513 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.519+0000 7ff4c4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:51.513 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.519+0000 7ff4c4ff9640 1 -- 172.21.15.114:0/784208792 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2300711997 0 0) 0x7ff4c80b2a10 con 0x7ff4c80b0860 2024-03-29T11:27:51.513 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.520+0000 7ff4c4ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:27:51.513 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.520+0000 7ff4c4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:51.513 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.520+0000 7ff4c4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:51.513 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.520+0000 7ff4c4ff9640 10 monclient(hunting): my global_id is 4499 2024-03-29T11:27:51.513 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.520+0000 7ff4c4ff9640 1 -- 172.21.15.114:0/784208792 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4b4004eb0 con 0x7ff4c80b0860 2024-03-29T11:27:51.513 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.520+0000 7ff4c4ff9640 1 -- 172.21.15.114:0/784208792 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2613628959 0 0) 0x7ff4b4004eb0 con 0x7ff4c80b0860 2024-03-29T11:27:51.514 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.520+0000 7ff4c4ff9640 1 -- 172.21.15.114:0/784208792 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff4c80b2a10 con 0x7ff4c80b0860 2024-03-29T11:27:51.514 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.520+0000 7ff4c4ff9640 1 -- 172.21.15.114:0/784208792 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7ff4c0001c70 con 0x7ff4c80b0860 2024-03-29T11:27:51.514 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.520+0000 7ff4c4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:51.514 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.520+0000 7ff4c4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:27:51.514 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.520+0000 7ff4c4ff9640 10 monclient(hunting): dump: 2024-03-29T11:27:51.514 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:27:51.514 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:27:51.514 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:27:51.514 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:27:51.514 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:51.514 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:27:51.514 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:27:51.514 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:27:51.514 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:27:51.515 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:27:51.515 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.520+0000 7ff4c4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:27:51.515 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.521+0000 7ff4c4ff9640 1 -- 172.21.15.114:0/784208792 <== mon.2 v1:172.21.15.114:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2219655726 0 0) 0x7ff4c80b2a10 con 0x7ff4c80b0860 2024-03-29T11:27:51.515 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.521+0000 7ff4c4ff9640 1 -- 172.21.15.114:0/784208792 >> v1:172.21.15.176:6789/0 conn(0x7ff4c806d9d0 legacy=0x7ff4c80b2300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:27:51.518 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.524+0000 7ff4c4ff9640 1 -- 172.21.15.114:0/784208792 >> v1:172.21.15.114:6789/0 conn(0x7ff4c813ef30 legacy=0x7ff4c81e6190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:51.518 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.524+0000 7ff4c4ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:27:51.518 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.524+0000 7ff4c4ff9640 1 monclient: found mon.c 2024-03-29T11:27:51.518 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.524+0000 7ff4c4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:51.518 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.524+0000 7ff4c4ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:27:51.518 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.524+0000 7ff4c4ff9640 1 -- 172.21.15.114:0/784208792 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4c80b2ba0 con 0x7ff4c80b0860 2024-03-29T11:27:51.519 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.525+0000 7ff4c4ff9640 10 monclient: _finish_auth 0 2024-03-29T11:27:51.519 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.525+0000 7ff4c4ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:27:51.519 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.525+0000 7ff4c4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:51.519 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.525+0000 7ff4c4ff9640 10 client.?.objecter ms_handle_connect 0x7ff4c813ef30 2024-03-29T11:27:51.519 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.525+0000 7ff4c4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:51.519 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.525+0000 7ff4c4ff9640 1 -- 172.21.15.114:0/784208792 <== mon.2 v1:172.21.15.114:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff4c0001dd0 con 0x7ff4c80b0860 2024-03-29T11:27:51.519 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.525+0000 7ff4c4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:51.519 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.526+0000 7ff4c67fc640 4 set_mon_vals no callback set 2024-03-29T11:27:51.519 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.526+0000 7ff4ce952640 5 monclient: authenticate success, global_id 4499 2024-03-29T11:27:51.519 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.526+0000 7ff4ce952640 10 monclient: _renew_subs 2024-03-29T11:27:51.520 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.526+0000 7ff4ce952640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:27:51.520 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.526+0000 7ff4ce952640 1 -- 172.21.15.114:0/784208792 --> v1:172.21.15.114:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4c80b1560 con 0x7ff4c80b0860 2024-03-29T11:27:51.520 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.527+0000 7ff4ce952640 10 client.4499.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:27:51.520 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.527+0000 7ff4ce952640 10 monclient: _renew_subs 2024-03-29T11:27:51.520 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.527+0000 7ff4ce952640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:27:51.520 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.527+0000 7ff4ce952640 1 -- 172.21.15.114:0/784208792 --> v1:172.21.15.114:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4c806df90 con 0x7ff4c80b0860 2024-03-29T11:27:51.520 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.527+0000 7ff4ce952640 1 librados: init done 2024-03-29T11:27:51.520 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.527+0000 7ff4c4ff9640 1 -- 172.21.15.114:0/784208792 <== mon.2 v1:172.21.15.114:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7ff4c0003c70 con 0x7ff4c80b0860 2024-03-29T11:27:51.520 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.527+0000 7ff4c4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:51.520 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.527+0000 7ff4c4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:51.520 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.527+0000 7ff4c4ff9640 10 monclient: dump: 2024-03-29T11:27:51.520 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:27:51.520 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:27:51.520 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:27:51.521 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:27:51.521 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:51.521 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:27:51.521 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:27:51.521 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:27:51.521 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:27:51.521 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:27:51.521 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.529+0000 7ff4ce952640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:51.521 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.529+0000 7ff4ce952640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:51.521 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.529+0000 7ff4ce952640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:27:51.521 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.529+0000 7ff4ce952640 1 -- 172.21.15.114:0/784208792 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4c80b39c0 con 0x7ff4c80b0860 2024-03-29T11:27:51.522 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.530+0000 7ff4c4ff9640 1 -- 172.21.15.114:0/784208792 <== mon.2 v1:172.21.15.114:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1594292461 0 0) 0x7ff4c0003dd0 con 0x7ff4c80b0860 2024-03-29T11:27:51.523 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.531+0000 7ff4c4ff9640 1 -- 172.21.15.114:0/784208792 <== mon.2 v1:172.21.15.114:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3238987313 0 0) 0x7ff4c0084df0 con 0x7ff4c80b0860 2024-03-29T11:27:51.523 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.531+0000 7ff4c4ff9640 10 client.4499.objecter ms_dispatch 0x7ff4c8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:27:51.523 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.531+0000 7ff4c4ff9640 3 client.4499.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:27:51.523 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.531+0000 7ff4c4ff9640 3 client.4499.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:27:51.523 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.531+0000 7ff4c4ff9640 20 client.4499.objecter dump_active .. 0 homeless 2024-03-29T11:27:51.523 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.532+0000 7ff4c7fff640 10 monclient: get_auth_request con 0x7ff4b406dc40 auth_method 0 2024-03-29T11:27:51.523 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.532+0000 7ff4c4ff9640 10 client.4499.objecter ms_handle_connect 0x7ff4b406dc40 2024-03-29T11:27:51.528 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.540+0000 7ff4c4ff9640 1 -- 172.21.15.114:0/784208792 <== mon.2 v1:172.21.15.114:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff4c0050500 con 0x7ff4c80b0860 2024-03-29T11:27:51.528 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.540+0000 7ff4c4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:51.529 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.540+0000 7ff4c4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:27:51.855 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.866+0000 7ff4ce952640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:51.855 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.866+0000 7ff4ce952640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:51.856 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.866+0000 7ff4ce952640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:27:51.856 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.866+0000 7ff4ce952640 1 -- 172.21.15.114:0/784208792 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff4c8137a10 con 0x7ff4c80b0860 2024-03-29T11:27:51.857 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.868+0000 7ff4c4ff9640 1 -- 172.21.15.114:0/784208792 <== mon.2 v1:172.21.15.114:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2858224613) 0x7ff4c0052600 con 0x7ff4c80b0860 2024-03-29T11:27:51.857 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.868+0000 7ff4c4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:51.857 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.868+0000 7ff4c4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:27:51.857 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T11:27:51.857 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"b0de2b9b-c45a-44c7-b6ee-e83b39a1e108","created":"2024-03-29T11:20:49.420660+0000","modified":"2024-03-29T11:20:51.700167+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:51.700167+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a64785d3-6b11-499a-b706-cc827ab96a74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5882dadb-e7a8-4604-ba9a-418013c106c8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d7e8c409-bc66-4e4a-bbdb-420bad31388e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0682989-3595-4cd6-90a5-ab9111c278e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"67068120-e3ff-4817-bd06-44feb7479a35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09e8b-8a85-4997-bdea-75d754e6ee67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2f0d9ead-f47f-4cb3-92a4-0a5ad23cd831","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fdb6a25f-6bd8-4c65-9b9b-b460ba7673a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:27:51.860 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.871+0000 7ff4ce952640 10 librados: watch_flush enter 2024-03-29T11:27:51.860 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.871+0000 7ff4ce952640 10 librados: watch_flush exit 2024-03-29T11:27:51.860 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.871+0000 7ff4ce952640 20 client.4499.objecter shutdown clearing up homeless session... 2024-03-29T11:27:51.860 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.871+0000 7ff4ce952640 10 client.4499.objecter successfully canceled tick 2024-03-29T11:27:51.861 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.872+0000 7ff4ce952640 1 -- 172.21.15.114:0/784208792 >> v1:172.21.15.114:6816/4038755055 conn(0x7ff4b406dc40 legacy=0x7ff4b4070100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:51.861 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.872+0000 7ff4ce952640 10 monclient: shutdown 2024-03-29T11:27:51.861 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.872+0000 7ff4ce952640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:51.861 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.872+0000 7ff4ce952640 1 -- 172.21.15.114:0/784208792 >> v1:172.21.15.114:6790/0 conn(0x7ff4c80b0860 legacy=0x7ff4c81362f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:51.861 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.872+0000 7ff4ce952640 1 -- 172.21.15.114:0/784208792 shutdown_connections 2024-03-29T11:27:51.861 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.872+0000 7ff4ce952640 1 -- 172.21.15.114:0/784208792 >> 172.21.15.114:0/784208792 conn(0x7ff4c806b660 msgr2=0x7ff4c806ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:51.861 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.872+0000 7ff4ce952640 1 -- 172.21.15.114:0/784208792 shutdown_connections 2024-03-29T11:27:51.861 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.872+0000 7ff4ce952640 1 -- 172.21.15.114:0/784208792 wait complete. 2024-03-29T11:27:51.861 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:51.872+0000 7ff4ce952640 1 librados: shutdown 2024-03-29T11:27:51.877 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:27:58.685 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:27:58.838 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.848+0000 7f50eaf4a640 10 monclient: get_monmap_and_config 2024-03-29T11:27:58.838 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.848+0000 7f50eaf4a640 10 monclient: build_initial_monmap 2024-03-29T11:27:58.838 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.848+0000 7f50eaf4a640 10 monclient: monmap: 2024-03-29T11:27:58.838 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:27:58.838 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:27:58.838 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:27:58.849796+0000 2024-03-29T11:27:58.838 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:27:58.849796+0000 2024-03-29T11:27:58.838 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:58.838 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:27:58.838 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:27:58.838 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T11:27:58.838 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T11:27:58.839 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:27:58.839 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.849+0000 7f50eaf4a640 1 Processor -- start 2024-03-29T11:27:58.839 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.849+0000 7f50eaf4a640 1 -- start start 2024-03-29T11:27:58.839 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.849+0000 7f50eaf4a640 10 monclient: init 2024-03-29T11:27:58.840 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.850+0000 7f50eaf4a640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:58.840 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.850+0000 7f50eaf4a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:27:58.840 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.850+0000 7f50eaf4a640 10 monclient(hunting): picked mon.noname-b con 0x7f50e40b4920 addr v1:172.21.15.114:6790/0 2024-03-29T11:27:58.840 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.850+0000 7f50eaf4a640 10 monclient(hunting): picked mon.noname-a con 0x7f50e40b4200 addr v1:172.21.15.114:6789/0 2024-03-29T11:27:58.840 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.850+0000 7f50eaf4a640 10 monclient(hunting): picked mon.noname-c con 0x7f50e413f9a0 addr v1:172.21.15.176:6789/0 2024-03-29T11:27:58.840 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.850+0000 7f50eaf4a640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50e40b0930 con 0x7f50e40b4200 2024-03-29T11:27:58.840 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.850+0000 7f50eaf4a640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50e40af460 con 0x7f50e413f9a0 2024-03-29T11:27:58.841 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.851+0000 7f50eaf4a640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50e40aeda0 con 0x7f50e40b4920 2024-03-29T11:27:58.841 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.851+0000 7f50eaf4a640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:58.841 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.851+0000 7f50eaf4a640 10 monclient(hunting): authenticate will time out at 1349.033112s 2024-03-29T11:27:58.842 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.852+0000 7f50e9f48640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f50e40b4920 0x7f50e413f290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:48540/0 (socket says 172.21.15.114:48540) 2024-03-29T11:27:58.842 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.852+0000 7f50e9f48640 1 -- 172.21.15.114:0/3247696603 learned_addr learned my addr 172.21.15.114:0/3247696603 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:27:58.842 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.853+0000 7f50e8f46640 1 -- 172.21.15.114:0/3247696603 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1583776554 0 0) 0x7f50e40aeda0 con 0x7f50e40b4920 2024-03-29T11:27:58.842 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.853+0000 7f50e8f46640 10 monclient(hunting): _negotiate 2024-03-29T11:27:58.843 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.853+0000 7f50e8f46640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:58.843 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.853+0000 7f50e8f46640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:58.843 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.853+0000 7f50e8f46640 10 monclient(hunting): my global_id is 4502 2024-03-29T11:27:58.843 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.853+0000 7f50e8f46640 1 -- 172.21.15.114:0/3247696603 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50d0003520 con 0x7f50e40b4920 2024-03-29T11:27:58.843 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.853+0000 7f50e8f46640 1 -- 172.21.15.114:0/3247696603 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1753926029 0 0) 0x7f50e40b0930 con 0x7f50e40b4200 2024-03-29T11:27:58.843 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.853+0000 7f50e8f46640 10 monclient(hunting): _negotiate 2024-03-29T11:27:58.843 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.853+0000 7f50e8f46640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:58.843 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.854+0000 7f50e8f46640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:58.843 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.854+0000 7f50e8f46640 10 monclient(hunting): my global_id is 4482 2024-03-29T11:27:58.843 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.854+0000 7f50e8f46640 1 -- 172.21.15.114:0/3247696603 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50e40aeda0 con 0x7f50e40b4200 2024-03-29T11:27:58.844 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.854+0000 7f50e8f46640 1 -- 172.21.15.114:0/3247696603 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2719327273 0 0) 0x7f50e40aeda0 con 0x7f50e40b4200 2024-03-29T11:27:58.844 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.854+0000 7f50e8f46640 1 -- 172.21.15.114:0/3247696603 >> v1:172.21.15.114:6790/0 conn(0x7f50e40b4920 legacy=0x7f50e413f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:58.844 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.854+0000 7f50e8f46640 1 -- 172.21.15.114:0/3247696603 >> v1:172.21.15.176:6789/0 conn(0x7f50e413f9a0 legacy=0x7f50e414d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:58.844 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.854+0000 7f50e8f46640 10 monclient: _finish_hunting 0 2024-03-29T11:27:58.844 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.854+0000 7f50e8f46640 1 monclient: found mon.noname-a 2024-03-29T11:27:58.844 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.854+0000 7f50e8f46640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:58.844 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.854+0000 7f50e8f46640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.114:6789/0 2024-03-29T11:27:58.844 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.854+0000 7f50e8f46640 1 -- 172.21.15.114:0/3247696603 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50e40af980 con 0x7f50e40b4200 2024-03-29T11:27:58.844 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.854+0000 7f50e8f46640 10 monclient: _finish_auth 0 2024-03-29T11:27:58.844 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.854+0000 7f50e8f46640 10 monclient: _check_auth_tickets 2024-03-29T11:27:58.845 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.854+0000 7f50e8f46640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:58.845 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.854+0000 7f50eaf4a640 5 monclient: authenticate success, global_id 4482 2024-03-29T11:27:58.845 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.854+0000 7f50eaf4a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:58.845 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.854+0000 7f50e8f46640 1 -- 172.21.15.114:0/3247696603 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f50d4001520 con 0x7f50e40b4200 2024-03-29T11:27:58.845 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.854+0000 7f50e8f46640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:58.845 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.854+0000 7f50e8f46640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:27:58.845 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.854+0000 7f50e8f46640 10 monclient: dump: 2024-03-29T11:27:58.845 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:27:58.845 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:27:58.845 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:27:58.845 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:27:58.845 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:58.845 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:27:58.845 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:27:58.845 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:27:58.846 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:27:58.846 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:27:58.846 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.854+0000 7f50eaf4a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:58.846 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.854+0000 7f50e8f46640 1 -- 172.21.15.114:0/3247696603 <== mon.0 v1:172.21.15.114:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f50d4001680 con 0x7f50e40b4200 2024-03-29T11:27:58.846 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.854+0000 7f50e8f46640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:58.847 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.854+0000 7f50eaf4a640 10 monclient: get_monmap_and_config success 2024-03-29T11:27:58.847 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.854+0000 7f50eaf4a640 4 set_mon_vals no callback set 2024-03-29T11:27:58.848 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.855+0000 7f50eaf4a640 10 monclient: shutdown 2024-03-29T11:27:58.848 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.855+0000 7f50eaf4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:58.848 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.855+0000 7f50eaf4a640 1 -- 172.21.15.114:0/3247696603 >> v1:172.21.15.114:6789/0 conn(0x7f50e40b4200 legacy=0x7f50e40b0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:58.848 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.856+0000 7f50eaf4a640 1 -- 172.21.15.114:0/3247696603 shutdown_connections 2024-03-29T11:27:58.848 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.856+0000 7f50eaf4a640 1 -- 172.21.15.114:0/3247696603 >> 172.21.15.114:0/3247696603 conn(0x7f50e406d230 msgr2=0x7f50e4066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:58.848 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.858+0000 7f50eaf4a640 1 -- 172.21.15.114:0/3247696603 shutdown_connections 2024-03-29T11:27:58.848 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.858+0000 7f50eaf4a640 1 -- 172.21.15.114:0/3247696603 wait complete. 2024-03-29T11:27:58.848 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.859+0000 7f50eaf4a640 10 monclient: build_initial_monmap 2024-03-29T11:27:58.848 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.859+0000 7f50eaf4a640 10 monclient: monmap: 2024-03-29T11:27:58.849 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:27:58.849 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:27:58.849 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T11:27:58.849 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T11:27:58.849 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:58.849 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:27:58.849 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:27:58.849 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T11:27:58.849 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T11:27:58.849 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:27:58.851 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.860+0000 7f50eaf4a640 1 librados: starting msgr at 2024-03-29T11:27:58.852 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.860+0000 7f50eaf4a640 1 librados: starting objecter 2024-03-29T11:27:58.852 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.860+0000 7f50eaf4a640 1 Processor -- start 2024-03-29T11:27:58.852 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.860+0000 7f50eaf4a640 1 -- start start 2024-03-29T11:27:58.852 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.860+0000 7f50eaf4a640 1 librados: setting wanted keys 2024-03-29T11:27:58.852 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.860+0000 7f50eaf4a640 1 librados: calling monclient init 2024-03-29T11:27:58.852 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.860+0000 7f50eaf4a640 10 monclient: init 2024-03-29T11:27:58.852 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.861+0000 7f50eaf4a640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:58.852 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.861+0000 7f50eaf4a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:27:58.852 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.861+0000 7f50eaf4a640 10 monclient(hunting): picked mon.noname-b con 0x7f50e40b4200 addr v1:172.21.15.176:6789/0 2024-03-29T11:27:58.852 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.861+0000 7f50eaf4a640 10 monclient(hunting): picked mon.noname-c con 0x7f50e413f9a0 addr v1:172.21.15.114:6790/0 2024-03-29T11:27:58.852 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.861+0000 7f50eaf4a640 10 monclient(hunting): picked mon.noname-a con 0x7f50e41dc030 addr v1:172.21.15.114:6789/0 2024-03-29T11:27:58.852 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.861+0000 7f50eaf4a640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50e406e660 con 0x7f50e41dc030 2024-03-29T11:27:58.852 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.861+0000 7f50eaf4a640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50e4145410 con 0x7f50e40b4200 2024-03-29T11:27:58.852 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.861+0000 7f50eaf4a640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50e41455a0 con 0x7f50e413f9a0 2024-03-29T11:27:58.852 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.861+0000 7f50eaf4a640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:58.853 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.861+0000 7f50eaf4a640 10 monclient(hunting): authenticate will time out at 1349.043493s 2024-03-29T11:27:58.854 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.865+0000 7f50e9747640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f50e413f9a0 0x7f50e41de960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:53376/0 (socket says 172.21.15.114:53376) 2024-03-29T11:27:58.854 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.865+0000 7f50e9747640 1 -- 172.21.15.114:0/1792534562 learned_addr learned my addr 172.21.15.114:0/1792534562 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:27:58.861 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.866+0000 7f50da7fc640 10 client.?.objecter ms_handle_connect 0x7f50e413f9a0 2024-03-29T11:27:58.861 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.866+0000 7f50da7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:58.861 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.866+0000 7f50da7fc640 1 -- 172.21.15.114:0/1792534562 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3109881832 0 0) 0x7f50e41455a0 con 0x7f50e413f9a0 2024-03-29T11:27:58.861 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.866+0000 7f50da7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:27:58.861 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.866+0000 7f50da7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:58.861 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.866+0000 7f50da7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:58.861 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.866+0000 7f50da7fc640 10 monclient(hunting): my global_id is 4505 2024-03-29T11:27:58.861 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.866+0000 7f50da7fc640 1 -- 172.21.15.114:0/1792534562 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50c8004eb0 con 0x7f50e413f9a0 2024-03-29T11:27:58.862 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.866+0000 7f50da7fc640 10 client.?.objecter ms_handle_connect 0x7f50e41dc030 2024-03-29T11:27:58.862 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.866+0000 7f50da7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:58.862 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.867+0000 7f50da7fc640 10 client.?.objecter ms_handle_connect 0x7f50e40b4200 2024-03-29T11:27:58.862 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.867+0000 7f50da7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:58.862 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.867+0000 7f50da7fc640 1 -- 172.21.15.114:0/1792534562 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3529888806 0 0) 0x7f50e4145410 con 0x7f50e40b4200 2024-03-29T11:27:58.862 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.867+0000 7f50da7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:27:58.862 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.867+0000 7f50da7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:58.862 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.867+0000 7f50da7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:58.862 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.867+0000 7f50da7fc640 10 monclient(hunting): my global_id is 4453 2024-03-29T11:27:58.862 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.867+0000 7f50da7fc640 1 -- 172.21.15.114:0/1792534562 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50e41455a0 con 0x7f50e40b4200 2024-03-29T11:27:58.862 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.867+0000 7f50da7fc640 1 -- 172.21.15.114:0/1792534562 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1110430726 0 0) 0x7f50e41455a0 con 0x7f50e40b4200 2024-03-29T11:27:58.862 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.867+0000 7f50da7fc640 1 -- 172.21.15.114:0/1792534562 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f50e4145410 con 0x7f50e40b4200 2024-03-29T11:27:58.862 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.867+0000 7f50da7fc640 1 -- 172.21.15.114:0/1792534562 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f50e0001950 con 0x7f50e40b4200 2024-03-29T11:27:58.862 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.867+0000 7f50da7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:58.863 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.867+0000 7f50da7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:27:58.863 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.867+0000 7f50da7fc640 10 monclient(hunting): dump: 2024-03-29T11:27:58.863 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:27:58.863 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:27:58.863 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:27:58.863 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:27:58.863 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:58.863 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:27:58.863 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:27:58.863 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:27:58.863 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:27:58.863 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:27:58.863 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.867+0000 7f50da7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:27:58.863 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.868+0000 7f50da7fc640 1 -- 172.21.15.114:0/1792534562 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2692925435 0 0) 0x7f50c8004eb0 con 0x7f50e413f9a0 2024-03-29T11:27:58.864 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.868+0000 7f50da7fc640 1 -- 172.21.15.114:0/1792534562 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f50e41455a0 con 0x7f50e413f9a0 2024-03-29T11:27:58.864 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.868+0000 7f50da7fc640 1 -- 172.21.15.114:0/1792534562 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f50d40035f0 con 0x7f50e413f9a0 2024-03-29T11:27:58.864 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.868+0000 7f50da7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:58.864 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.868+0000 7f50da7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:58.864 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.868+0000 7f50da7fc640 10 monclient(hunting): dump: 2024-03-29T11:27:58.864 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:27:58.864 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:27:58.864 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:27:58.864 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:27:58.864 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:58.864 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:27:58.864 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:27:58.864 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:27:58.864 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:27:58.864 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:27:58.864 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.868+0000 7f50da7fc640 1 -- 172.21.15.114:0/1792534562 <== mon.2 v1:172.21.15.114:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1794055436 0 0) 0x7f50e41455a0 con 0x7f50e413f9a0 2024-03-29T11:27:58.865 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.869+0000 7f50da7fc640 1 -- 172.21.15.114:0/1792534562 >> v1:172.21.15.176:6789/0 conn(0x7f50e40b4200 legacy=0x7f50e41db920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:58.865 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.869+0000 7f50da7fc640 1 -- 172.21.15.114:0/1792534562 >> v1:172.21.15.114:6789/0 conn(0x7f50e41dc030 legacy=0x7f50e41df070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:58.865 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.869+0000 7f50da7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:27:58.865 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.869+0000 7f50da7fc640 1 monclient: found mon.c 2024-03-29T11:27:58.865 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.869+0000 7f50da7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:58.865 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.869+0000 7f50da7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:27:58.865 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.869+0000 7f50da7fc640 1 -- 172.21.15.114:0/1792534562 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50e4144b90 con 0x7f50e413f9a0 2024-03-29T11:27:58.865 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.869+0000 7f50da7fc640 10 monclient: _finish_auth 0 2024-03-29T11:27:58.865 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.869+0000 7f50da7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:27:58.865 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.869+0000 7f50da7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:58.865 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.870+0000 7f50da7fc640 1 -- 172.21.15.114:0/1792534562 <== mon.2 v1:172.21.15.114:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f50d4003a80 con 0x7f50e413f9a0 2024-03-29T11:27:58.865 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.870+0000 7f50da7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:58.865 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.870+0000 7f50da7fc640 1 -- 172.21.15.114:0/1792534562 <== mon.2 v1:172.21.15.114:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f50d4003e10 con 0x7f50e413f9a0 2024-03-29T11:27:58.865 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.870+0000 7f50eaf4a640 5 monclient: authenticate success, global_id 4505 2024-03-29T11:27:58.865 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.870+0000 7f50eaf4a640 10 monclient: _renew_subs 2024-03-29T11:27:58.866 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.870+0000 7f50eaf4a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:27:58.866 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.870+0000 7f50eaf4a640 1 -- 172.21.15.114:0/1792534562 --> v1:172.21.15.114:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50e4144d80 con 0x7f50e413f9a0 2024-03-29T11:27:58.866 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.870+0000 7f50eaf4a640 10 client.4505.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:27:58.866 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.870+0000 7f50eaf4a640 10 monclient: _renew_subs 2024-03-29T11:27:58.866 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.870+0000 7f50eaf4a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:27:58.866 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.870+0000 7f50eaf4a640 1 -- 172.21.15.114:0/1792534562 --> v1:172.21.15.114:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f50e4142360 con 0x7f50e413f9a0 2024-03-29T11:27:58.866 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.870+0000 7f50eaf4a640 1 librados: init done 2024-03-29T11:27:58.866 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.870+0000 7f50da7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:58.866 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.871+0000 7f50da7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:58.866 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.871+0000 7f50da7fc640 10 monclient: dump: 2024-03-29T11:27:58.866 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:27:58.866 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:27:58.866 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:27:58.866 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:27:58.867 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:58.867 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:27:58.867 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:27:58.867 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:27:58.867 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:27:58.867 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:27:58.868 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.871+0000 7f50dbfff640 4 set_mon_vals no callback set 2024-03-29T11:27:58.868 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.873+0000 7f50eaf4a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:58.868 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.873+0000 7f50eaf4a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:58.868 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.873+0000 7f50eaf4a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:27:58.868 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.874+0000 7f50eaf4a640 1 -- 172.21.15.114:0/1792534562 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50cc005080 con 0x7f50e413f9a0 2024-03-29T11:27:58.868 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.877+0000 7f50da7fc640 1 -- 172.21.15.114:0/1792534562 <== mon.2 v1:172.21.15.114:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1594292461 0 0) 0x7f50d40018b0 con 0x7f50e413f9a0 2024-03-29T11:27:58.869 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.878+0000 7f50da7fc640 1 -- 172.21.15.114:0/1792534562 <== mon.2 v1:172.21.15.114:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3238987313 0 0) 0x7f50d4084a80 con 0x7f50e413f9a0 2024-03-29T11:27:58.869 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.880+0000 7f50da7fc640 10 client.4505.objecter ms_dispatch 0x7f50e4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:27:58.869 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.880+0000 7f50da7fc640 3 client.4505.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:27:58.869 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.880+0000 7f50da7fc640 3 client.4505.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:27:58.869 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.880+0000 7f50da7fc640 20 client.4505.objecter dump_active .. 0 homeless 2024-03-29T11:27:58.870 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.880+0000 7f50e9f48640 10 monclient: get_auth_request con 0x7f50c806deb0 auth_method 0 2024-03-29T11:27:58.871 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.882+0000 7f50da7fc640 10 client.4505.objecter ms_handle_connect 0x7f50c806deb0 2024-03-29T11:27:58.873 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.884+0000 7f50da7fc640 1 -- 172.21.15.114:0/1792534562 <== mon.2 v1:172.21.15.114:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f50d4050360 con 0x7f50e413f9a0 2024-03-29T11:27:58.873 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.884+0000 7f50da7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:58.873 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:58.884+0000 7f50da7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:27:59.189 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:59.200+0000 7f50eaf4a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:59.190 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:59.200+0000 7f50eaf4a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:59.190 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:59.200+0000 7f50eaf4a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:27:59.190 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:59.200+0000 7f50eaf4a640 1 -- 172.21.15.114:0/1792534562 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f50cc005330 con 0x7f50e413f9a0 2024-03-29T11:27:59.190 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:59.201+0000 7f50da7fc640 1 -- 172.21.15.114:0/1792534562 <== mon.2 v1:172.21.15.114:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2858224613) 0x7f50d4052290 con 0x7f50e413f9a0 2024-03-29T11:27:59.190 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:59.201+0000 7f50da7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:59.191 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:59.201+0000 7f50da7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:27:59.193 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T11:27:59.194 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"b0de2b9b-c45a-44c7-b6ee-e83b39a1e108","created":"2024-03-29T11:20:49.420660+0000","modified":"2024-03-29T11:20:51.700167+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:51.700167+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a64785d3-6b11-499a-b706-cc827ab96a74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5882dadb-e7a8-4604-ba9a-418013c106c8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d7e8c409-bc66-4e4a-bbdb-420bad31388e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0682989-3595-4cd6-90a5-ab9111c278e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"67068120-e3ff-4817-bd06-44feb7479a35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09e8b-8a85-4997-bdea-75d754e6ee67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2f0d9ead-f47f-4cb3-92a4-0a5ad23cd831","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fdb6a25f-6bd8-4c65-9b9b-b460ba7673a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:27:59.197 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:59.208+0000 7f50eaf4a640 10 librados: watch_flush enter 2024-03-29T11:27:59.197 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:59.208+0000 7f50eaf4a640 10 librados: watch_flush exit 2024-03-29T11:27:59.197 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:59.208+0000 7f50eaf4a640 20 client.4505.objecter shutdown clearing up homeless session... 2024-03-29T11:27:59.197 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:59.208+0000 7f50eaf4a640 10 client.4505.objecter successfully canceled tick 2024-03-29T11:27:59.199 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:59.210+0000 7f50eaf4a640 1 -- 172.21.15.114:0/1792534562 >> v1:172.21.15.114:6816/4038755055 conn(0x7f50c806deb0 legacy=0x7f50c8070370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:59.199 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:59.210+0000 7f50eaf4a640 10 monclient: shutdown 2024-03-29T11:27:59.199 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:59.210+0000 7f50eaf4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:59.199 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:59.210+0000 7f50eaf4a640 1 -- 172.21.15.114:0/1792534562 >> v1:172.21.15.114:6790/0 conn(0x7f50e413f9a0 legacy=0x7f50e41de960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:59.199 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:59.210+0000 7f50eaf4a640 1 -- 172.21.15.114:0/1792534562 shutdown_connections 2024-03-29T11:27:59.199 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:59.210+0000 7f50eaf4a640 1 -- 172.21.15.114:0/1792534562 >> 172.21.15.114:0/1792534562 conn(0x7f50e406d230 msgr2=0x7f50e4066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:59.200 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:59.210+0000 7f50eaf4a640 1 -- 172.21.15.114:0/1792534562 shutdown_connections 2024-03-29T11:27:59.200 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:59.210+0000 7f50eaf4a640 1 -- 172.21.15.114:0/1792534562 wait complete. 2024-03-29T11:27:59.200 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:27:59.210+0000 7f50eaf4a640 1 librados: shutdown 2024-03-29T11:27:59.215 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:28:06.022 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:28:06.174 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.183+0000 7f2ed5f4d640 10 monclient: get_monmap_and_config 2024-03-29T11:28:06.174 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.183+0000 7f2ed5f4d640 10 monclient: build_initial_monmap 2024-03-29T11:28:06.174 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.183+0000 7f2ed5f4d640 10 monclient: monmap: 2024-03-29T11:28:06.174 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:28:06.174 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:28:06.174 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:28:06.184912+0000 2024-03-29T11:28:06.174 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:28:06.184912+0000 2024-03-29T11:28:06.175 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:06.175 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:28:06.175 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:28:06.175 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T11:28:06.175 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T11:28:06.175 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:28:06.175 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.185+0000 7f2ed5f4d640 1 Processor -- start 2024-03-29T11:28:06.175 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.185+0000 7f2ed5f4d640 1 -- start start 2024-03-29T11:28:06.175 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.185+0000 7f2ed5f4d640 10 monclient: init 2024-03-29T11:28:06.186 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.186+0000 7f2ed5f4d640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:06.186 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.186+0000 7f2ed5f4d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:28:06.186 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.186+0000 7f2ed5f4d640 10 monclient(hunting): picked mon.noname-c con 0x7f2ec8006410 addr v1:172.21.15.176:6789/0 2024-03-29T11:28:06.187 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.186+0000 7f2ed5f4d640 10 monclient(hunting): picked mon.noname-b con 0x7f2ec802a640 addr v1:172.21.15.114:6790/0 2024-03-29T11:28:06.187 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.186+0000 7f2ed5f4d640 10 monclient(hunting): picked mon.noname-a con 0x7f2ec80327d0 addr v1:172.21.15.114:6789/0 2024-03-29T11:28:06.187 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.186+0000 7f2ed5f4d640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ec802c820 con 0x7f2ec80327d0 2024-03-29T11:28:06.187 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.186+0000 7f2ed5f4d640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ec802a120 con 0x7f2ec8006410 2024-03-29T11:28:06.187 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.186+0000 7f2ed5f4d640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ec80c75b0 con 0x7f2ec802a640 2024-03-29T11:28:06.187 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.186+0000 7f2ed5f4d640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:06.187 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.186+0000 7f2ed5f4d640 10 monclient(hunting): authenticate will time out at 1356.368623s 2024-03-29T11:28:06.187 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.193+0000 7f2ed4f4b640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f2ec8006410 0x7f2ec80c6ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.114:45277/0 (socket says 172.21.15.114:45277) 2024-03-29T11:28:06.187 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.193+0000 7f2ed4f4b640 1 -- 172.21.15.114:0/2886200938 learned_addr learned my addr 172.21.15.114:0/2886200938 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:28:06.187 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.193+0000 7f2ecf7fe640 1 -- 172.21.15.114:0/2886200938 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3374932733 0 0) 0x7f2ec80c75b0 con 0x7f2ec802a640 2024-03-29T11:28:06.187 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.193+0000 7f2ecf7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:28:06.187 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.193+0000 7f2ecf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:06.188 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.193+0000 7f2ecf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:06.188 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.193+0000 7f2ecf7fe640 10 monclient(hunting): my global_id is 4508 2024-03-29T11:28:06.188 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.193+0000 7f2ecf7fe640 1 -- 172.21.15.114:0/2886200938 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ec0003520 con 0x7f2ec802a640 2024-03-29T11:28:06.188 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.194+0000 7f2ecf7fe640 1 -- 172.21.15.114:0/2886200938 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2595395563 0 0) 0x7f2ec802a120 con 0x7f2ec8006410 2024-03-29T11:28:06.188 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.194+0000 7f2ecf7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:28:06.188 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.194+0000 7f2ecf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:06.188 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.194+0000 7f2ecf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:06.188 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.194+0000 7f2ecf7fe640 10 monclient(hunting): my global_id is 4456 2024-03-29T11:28:06.188 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.194+0000 7f2ecf7fe640 1 -- 172.21.15.114:0/2886200938 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ec80c75b0 con 0x7f2ec8006410 2024-03-29T11:28:06.188 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.194+0000 7f2ecf7fe640 1 -- 172.21.15.114:0/2886200938 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 372427689 0 0) 0x7f2ec802c820 con 0x7f2ec80327d0 2024-03-29T11:28:06.188 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.194+0000 7f2ecf7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:28:06.188 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.194+0000 7f2ecf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:06.188 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.194+0000 7f2ecf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:06.188 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.194+0000 7f2ecf7fe640 10 monclient(hunting): my global_id is 4485 2024-03-29T11:28:06.189 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.194+0000 7f2ecf7fe640 1 -- 172.21.15.114:0/2886200938 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ec802a120 con 0x7f2ec80327d0 2024-03-29T11:28:06.189 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.194+0000 7f2ecf7fe640 1 -- 172.21.15.114:0/2886200938 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4189610012 0 0) 0x7f2ec80c75b0 con 0x7f2ec8006410 2024-03-29T11:28:06.189 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.194+0000 7f2ecf7fe640 1 -- 172.21.15.114:0/2886200938 >> v1:172.21.15.114:6790/0 conn(0x7f2ec802a640 legacy=0x7f2ec80320c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:06.189 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.194+0000 7f2ecf7fe640 1 -- 172.21.15.114:0/2886200938 >> v1:172.21.15.114:6789/0 conn(0x7f2ec80327d0 legacy=0x7f2ec80c8200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:06.189 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.194+0000 7f2ecf7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:28:06.189 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.194+0000 7f2ecf7fe640 1 monclient: found mon.noname-c 2024-03-29T11:28:06.189 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.194+0000 7f2ecf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:06.189 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.194+0000 7f2ecf7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.176:6789/0 2024-03-29T11:28:06.189 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.195+0000 7f2ecf7fe640 1 -- 172.21.15.114:0/2886200938 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ec8033110 con 0x7f2ec8006410 2024-03-29T11:28:06.189 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.195+0000 7f2ecf7fe640 10 monclient: _finish_auth 0 2024-03-29T11:28:06.189 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.195+0000 7f2ecf7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:28:06.189 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.195+0000 7f2ecf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:06.189 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.195+0000 7f2ed5f4d640 5 monclient: authenticate success, global_id 4456 2024-03-29T11:28:06.190 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.195+0000 7f2ed5f4d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:06.190 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.195+0000 7f2ecf7fe640 1 -- 172.21.15.114:0/2886200938 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f2ec4001670 con 0x7f2ec8006410 2024-03-29T11:28:06.190 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.195+0000 7f2ecf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:06.190 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.195+0000 7f2ecf7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:28:06.190 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.195+0000 7f2ecf7fe640 10 monclient: dump: 2024-03-29T11:28:06.190 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:28:06.190 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:28:06.190 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:28:06.190 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:28:06.190 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:06.190 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:28:06.190 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:28:06.190 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:28:06.190 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:28:06.190 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:28:06.191 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.195+0000 7f2ed5f4d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:06.193 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.198+0000 7f2ecf7fe640 1 -- 172.21.15.114:0/2886200938 <== mon.1 v1:172.21.15.176:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2ec4001840 con 0x7f2ec8006410 2024-03-29T11:28:06.193 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.198+0000 7f2ecf7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:06.193 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.198+0000 7f2ed5f4d640 10 monclient: get_monmap_and_config success 2024-03-29T11:28:06.193 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.198+0000 7f2ed5f4d640 4 set_mon_vals no callback set 2024-03-29T11:28:06.193 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.198+0000 7f2ecf7fe640 1 -- 172.21.15.114:0/2886200938 <== mon.1 v1:172.21.15.176:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f2ec4001bd0 con 0x7f2ec8006410 2024-03-29T11:28:06.193 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.198+0000 7f2ecf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:06.193 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.198+0000 7f2ecf7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:28:06.193 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.198+0000 7f2ecf7fe640 10 monclient: dump: 2024-03-29T11:28:06.193 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:28:06.193 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:28:06.194 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:28:06.194 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:28:06.194 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:06.194 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:28:06.194 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:28:06.194 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:28:06.194 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:28:06.194 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:28:06.194 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.199+0000 7f2ed5f4d640 10 monclient: shutdown 2024-03-29T11:28:06.194 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.199+0000 7f2ed5f4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:06.194 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.199+0000 7f2ed5f4d640 1 -- 172.21.15.114:0/2886200938 >> v1:172.21.15.176:6789/0 conn(0x7f2ec8006410 legacy=0x7f2ec80c6ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:06.194 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.199+0000 7f2ed5f4d640 1 -- 172.21.15.114:0/2886200938 shutdown_connections 2024-03-29T11:28:06.194 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.199+0000 7f2ed5f4d640 1 -- 172.21.15.114:0/2886200938 >> 172.21.15.114:0/2886200938 conn(0x7f2ec80058f0 msgr2=0x7f2ec8005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:06.194 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.199+0000 7f2ed5f4d640 1 -- 172.21.15.114:0/2886200938 shutdown_connections 2024-03-29T11:28:06.194 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.199+0000 7f2ed5f4d640 1 -- 172.21.15.114:0/2886200938 wait complete. 2024-03-29T11:28:06.195 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.200+0000 7f2ed5f4d640 10 monclient: build_initial_monmap 2024-03-29T11:28:06.195 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.200+0000 7f2ed5f4d640 10 monclient: monmap: 2024-03-29T11:28:06.195 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:28:06.195 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:28:06.195 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T11:28:06.195 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T11:28:06.195 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:06.195 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:28:06.195 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:28:06.195 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T11:28:06.195 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T11:28:06.195 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:28:06.195 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.200+0000 7f2ed5f4d640 1 librados: starting msgr at 2024-03-29T11:28:06.196 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.200+0000 7f2ed5f4d640 1 librados: starting objecter 2024-03-29T11:28:06.196 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.200+0000 7f2ed5f4d640 1 Processor -- start 2024-03-29T11:28:06.196 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.201+0000 7f2ed5f4d640 1 -- start start 2024-03-29T11:28:06.196 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.201+0000 7f2ed5f4d640 1 librados: setting wanted keys 2024-03-29T11:28:06.196 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.201+0000 7f2ed5f4d640 1 librados: calling monclient init 2024-03-29T11:28:06.196 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.201+0000 7f2ed5f4d640 10 monclient: init 2024-03-29T11:28:06.196 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.202+0000 7f2ed5f4d640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:06.196 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.202+0000 7f2ed5f4d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:28:06.196 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.202+0000 7f2ed5f4d640 10 monclient(hunting): picked mon.noname-c con 0x7f2ec8006410 addr v1:172.21.15.114:6790/0 2024-03-29T11:28:06.196 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.202+0000 7f2ed5f4d640 10 monclient(hunting): picked mon.noname-a con 0x7f2ec802a640 addr v1:172.21.15.114:6789/0 2024-03-29T11:28:06.196 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.202+0000 7f2ed5f4d640 10 monclient(hunting): picked mon.noname-b con 0x7f2ec80327d0 addr v1:172.21.15.176:6789/0 2024-03-29T11:28:06.196 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.202+0000 7f2ed5f4d640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ec80d65a0 con 0x7f2ec802a640 2024-03-29T11:28:06.196 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.202+0000 7f2ed5f4d640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ec80d70f0 con 0x7f2ec80327d0 2024-03-29T11:28:06.197 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.202+0000 7f2ed5f4d640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ec80d73d0 con 0x7f2ec8006410 2024-03-29T11:28:06.197 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.202+0000 7f2ed5f4d640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:06.197 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.202+0000 7f2ed5f4d640 10 monclient(hunting): authenticate will time out at 1356.384291s 2024-03-29T11:28:06.202 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.206+0000 7f2ecffff640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f2ec802a640 0x7f2ec80bc860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:36439/0 (socket says 172.21.15.114:36439) 2024-03-29T11:28:06.202 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.206+0000 7f2ecffff640 1 -- 172.21.15.114:0/2980406378 learned_addr learned my addr 172.21.15.114:0/2980406378 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:28:06.202 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.206+0000 7f2ecd7fa640 10 client.?.objecter ms_handle_connect 0x7f2ec802a640 2024-03-29T11:28:06.202 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.206+0000 7f2ecd7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:06.203 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.206+0000 7f2ecd7fa640 10 client.?.objecter ms_handle_connect 0x7f2ec8006410 2024-03-29T11:28:06.203 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.206+0000 7f2ecd7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:06.203 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.207+0000 7f2ecd7fa640 1 -- 172.21.15.114:0/2980406378 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1127952806 0 0) 0x7f2ec80d73d0 con 0x7f2ec8006410 2024-03-29T11:28:06.203 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.207+0000 7f2ecd7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:28:06.203 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.207+0000 7f2ecd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:06.203 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.207+0000 7f2ecd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:06.203 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.207+0000 7f2ecd7fa640 10 monclient(hunting): my global_id is 4511 2024-03-29T11:28:06.203 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.207+0000 7f2ecd7fa640 1 -- 172.21.15.114:0/2980406378 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2eb4004eb0 con 0x7f2ec8006410 2024-03-29T11:28:06.203 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.207+0000 7f2ecd7fa640 1 -- 172.21.15.114:0/2980406378 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2739490217 0 0) 0x7f2ec80d65a0 con 0x7f2ec802a640 2024-03-29T11:28:06.203 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.207+0000 7f2ecd7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:28:06.203 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.207+0000 7f2ecd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:06.203 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.207+0000 7f2ecd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:06.203 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.207+0000 7f2ecd7fa640 10 monclient(hunting): my global_id is 4488 2024-03-29T11:28:06.204 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.207+0000 7f2ecd7fa640 1 -- 172.21.15.114:0/2980406378 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ec80d73d0 con 0x7f2ec802a640 2024-03-29T11:28:06.204 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.208+0000 7f2ecd7fa640 1 -- 172.21.15.114:0/2980406378 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 687238899 0 0) 0x7f2ec80d73d0 con 0x7f2ec802a640 2024-03-29T11:28:06.204 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.208+0000 7f2ecd7fa640 1 -- 172.21.15.114:0/2980406378 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2ec80d65a0 con 0x7f2ec802a640 2024-03-29T11:28:06.204 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.208+0000 7f2ecd7fa640 1 -- 172.21.15.114:0/2980406378 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f2ebc001a70 con 0x7f2ec802a640 2024-03-29T11:28:06.204 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.208+0000 7f2ecd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:06.204 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.208+0000 7f2ecd7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:28:06.204 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.208+0000 7f2ecd7fa640 10 monclient(hunting): dump: 2024-03-29T11:28:06.204 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:28:06.204 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:28:06.204 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:28:06.204 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:28:06.204 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:06.204 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:28:06.204 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:28:06.205 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:28:06.205 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:28:06.205 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:28:06.205 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.208+0000 7f2ecd7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:28:06.205 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.209+0000 7f2ecd7fa640 1 -- 172.21.15.114:0/2980406378 <== mon.0 v1:172.21.15.114:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1443619394 0 0) 0x7f2ec80d65a0 con 0x7f2ec802a640 2024-03-29T11:28:06.205 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.209+0000 7f2ecd7fa640 1 -- 172.21.15.114:0/2980406378 >> v1:172.21.15.114:6790/0 conn(0x7f2ec8006410 legacy=0x7f2ec80c3710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:06.205 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.209+0000 7f2ecd7fa640 1 -- 172.21.15.114:0/2980406378 >> v1:172.21.15.176:6789/0 conn(0x7f2ec80327d0 legacy=0x7f2ec80bcf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:06.205 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.209+0000 7f2ecd7fa640 10 monclient: _finish_hunting 0 2024-03-29T11:28:06.205 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.209+0000 7f2ecd7fa640 1 monclient: found mon.a 2024-03-29T11:28:06.205 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.209+0000 7f2ecd7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:06.205 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.209+0000 7f2ecd7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T11:28:06.205 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.209+0000 7f2ecd7fa640 1 -- 172.21.15.114:0/2980406378 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ec80d76b0 con 0x7f2ec802a640 2024-03-29T11:28:06.205 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.209+0000 7f2ecd7fa640 10 monclient: _finish_auth 0 2024-03-29T11:28:06.205 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.209+0000 7f2ecd7fa640 10 monclient: _check_auth_tickets 2024-03-29T11:28:06.205 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.209+0000 7f2ecd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:06.206 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.210+0000 7f2ed5f4d640 5 monclient: authenticate success, global_id 4488 2024-03-29T11:28:06.206 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.210+0000 7f2ed5f4d640 10 monclient: _renew_subs 2024-03-29T11:28:06.206 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.210+0000 7f2ed5f4d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T11:28:06.206 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.210+0000 7f2ed5f4d640 1 -- 172.21.15.114:0/2980406378 --> v1:172.21.15.114:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ec80d7990 con 0x7f2ec802a640 2024-03-29T11:28:06.206 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.210+0000 7f2ed5f4d640 10 client.4488.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:28:06.206 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.210+0000 7f2ed5f4d640 10 monclient: _renew_subs 2024-03-29T11:28:06.206 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.210+0000 7f2ed5f4d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T11:28:06.206 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.210+0000 7f2ed5f4d640 1 -- 172.21.15.114:0/2980406378 --> v1:172.21.15.114:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ec816c760 con 0x7f2ec802a640 2024-03-29T11:28:06.206 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.210+0000 7f2ed5f4d640 1 librados: init done 2024-03-29T11:28:06.206 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.210+0000 7f2ecd7fa640 1 -- 172.21.15.114:0/2980406378 <== mon.0 v1:172.21.15.114:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2ebc001c90 con 0x7f2ec802a640 2024-03-29T11:28:06.206 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.210+0000 7f2ecd7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:06.206 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.210+0000 7f2eceffd640 4 set_mon_vals no callback set 2024-03-29T11:28:06.206 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.212+0000 7f2ecd7fa640 1 -- 172.21.15.114:0/2980406378 <== mon.0 v1:172.21.15.114:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f2ebc003a40 con 0x7f2ec802a640 2024-03-29T11:28:06.206 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.212+0000 7f2ecd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:06.206 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.212+0000 7f2ecd7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:28:06.207 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.212+0000 7f2ecd7fa640 10 monclient: dump: 2024-03-29T11:28:06.207 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:28:06.207 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:28:06.207 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:28:06.207 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:28:06.207 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:06.207 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:28:06.207 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:28:06.207 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:28:06.207 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:28:06.207 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:28:06.207 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.212+0000 7f2ed5f4d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:06.207 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.212+0000 7f2ed5f4d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:06.207 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.212+0000 7f2ed5f4d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T11:28:06.207 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.212+0000 7f2ed5f4d640 1 -- 172.21.15.114:0/2980406378 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2eb0005080 con 0x7f2ec802a640 2024-03-29T11:28:06.209 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.214+0000 7f2ecd7fa640 1 -- 172.21.15.114:0/2980406378 <== mon.0 v1:172.21.15.114:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1594292461 0 0) 0x7f2ebc003c20 con 0x7f2ec802a640 2024-03-29T11:28:06.209 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.215+0000 7f2ecd7fa640 1 -- 172.21.15.114:0/2980406378 <== mon.0 v1:172.21.15.114:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3238987313 0 0) 0x7f2ebc054900 con 0x7f2ec802a640 2024-03-29T11:28:06.209 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.215+0000 7f2ecd7fa640 10 client.4488.objecter ms_dispatch 0x7f2ed0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:28:06.209 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.215+0000 7f2ecd7fa640 3 client.4488.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:28:06.209 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.215+0000 7f2ecd7fa640 3 client.4488.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:28:06.209 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.215+0000 7f2ecd7fa640 20 client.4488.objecter dump_active .. 0 homeless 2024-03-29T11:28:06.209 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.215+0000 7f2ed4f4b640 10 monclient: get_auth_request con 0x7f2eb406dc90 auth_method 0 2024-03-29T11:28:06.209 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.216+0000 7f2ecd7fa640 10 client.4488.objecter ms_handle_connect 0x7f2eb406dc90 2024-03-29T11:28:06.214 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.223+0000 7f2ecd7fa640 1 -- 172.21.15.114:0/2980406378 <== mon.0 v1:172.21.15.114:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2ebc050ba0 con 0x7f2ec802a640 2024-03-29T11:28:06.214 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.223+0000 7f2ecd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:06.214 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.223+0000 7f2ecd7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:28:06.548 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.558+0000 7f2ed5f4d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:06.548 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.558+0000 7f2ed5f4d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:06.549 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.558+0000 7f2ed5f4d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T11:28:06.549 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.558+0000 7f2ed5f4d640 1 -- 172.21.15.114:0/2980406378 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2eb0005330 con 0x7f2ec802a640 2024-03-29T11:28:06.550 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T11:28:06.550 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"b0de2b9b-c45a-44c7-b6ee-e83b39a1e108","created":"2024-03-29T11:20:49.420660+0000","modified":"2024-03-29T11:20:51.700167+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:51.700167+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a64785d3-6b11-499a-b706-cc827ab96a74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5882dadb-e7a8-4604-ba9a-418013c106c8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d7e8c409-bc66-4e4a-bbdb-420bad31388e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0682989-3595-4cd6-90a5-ab9111c278e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"67068120-e3ff-4817-bd06-44feb7479a35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09e8b-8a85-4997-bdea-75d754e6ee67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2f0d9ead-f47f-4cb3-92a4-0a5ad23cd831","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fdb6a25f-6bd8-4c65-9b9b-b460ba7673a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:28:06.550 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.558+0000 7f2ecd7fa640 1 -- 172.21.15.114:0/2980406378 <== mon.0 v1:172.21.15.114:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2858224613) 0x7f2ebc050ba0 con 0x7f2ec802a640 2024-03-29T11:28:06.550 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.559+0000 7f2ecd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:06.550 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.559+0000 7f2ecd7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:28:06.551 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.562+0000 7f2ed5f4d640 10 librados: watch_flush enter 2024-03-29T11:28:06.552 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.562+0000 7f2ed5f4d640 10 librados: watch_flush exit 2024-03-29T11:28:06.552 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.562+0000 7f2ed5f4d640 20 client.4488.objecter shutdown clearing up homeless session... 2024-03-29T11:28:06.552 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.562+0000 7f2ed5f4d640 10 client.4488.objecter successfully canceled tick 2024-03-29T11:28:06.552 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.563+0000 7f2ed5f4d640 1 -- 172.21.15.114:0/2980406378 >> v1:172.21.15.114:6816/4038755055 conn(0x7f2eb406dc90 legacy=0x7f2eb4070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:06.553 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.563+0000 7f2ed5f4d640 10 monclient: shutdown 2024-03-29T11:28:06.553 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.563+0000 7f2ed5f4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:06.553 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.563+0000 7f2ed5f4d640 1 -- 172.21.15.114:0/2980406378 >> v1:172.21.15.114:6789/0 conn(0x7f2ec802a640 legacy=0x7f2ec80bc860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:06.554 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.565+0000 7f2ed5f4d640 1 -- 172.21.15.114:0/2980406378 shutdown_connections 2024-03-29T11:28:06.554 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.565+0000 7f2ed5f4d640 1 -- 172.21.15.114:0/2980406378 >> 172.21.15.114:0/2980406378 conn(0x7f2ec80058f0 msgr2=0x7f2ec80043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:06.554 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.565+0000 7f2ed5f4d640 1 -- 172.21.15.114:0/2980406378 shutdown_connections 2024-03-29T11:28:06.554 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.565+0000 7f2ed5f4d640 1 -- 172.21.15.114:0/2980406378 wait complete. 2024-03-29T11:28:06.554 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:06.565+0000 7f2ed5f4d640 1 librados: shutdown 2024-03-29T11:28:06.572 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:28:13.383 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:28:13.542 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.553+0000 7f042ded6640 10 monclient: get_monmap_and_config 2024-03-29T11:28:13.542 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.553+0000 7f042ded6640 10 monclient: build_initial_monmap 2024-03-29T11:28:13.543 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.553+0000 7f042ded6640 10 monclient: monmap: 2024-03-29T11:28:13.543 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:28:13.543 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:28:13.543 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:28:13.554092+0000 2024-03-29T11:28:13.543 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:28:13.554092+0000 2024-03-29T11:28:13.543 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:13.543 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:28:13.543 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:28:13.543 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T11:28:13.543 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T11:28:13.543 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:28:13.544 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.554+0000 7f042ded6640 1 Processor -- start 2024-03-29T11:28:13.544 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.554+0000 7f042ded6640 1 -- start start 2024-03-29T11:28:13.544 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.554+0000 7f042ded6640 10 monclient: init 2024-03-29T11:28:13.547 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.555+0000 7f042ded6640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:13.547 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.555+0000 7f042ded6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:28:13.547 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.555+0000 7f042ded6640 10 monclient(hunting): picked mon.noname-c con 0x7f04281422b0 addr v1:172.21.15.176:6789/0 2024-03-29T11:28:13.547 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.555+0000 7f042ded6640 10 monclient(hunting): picked mon.noname-a con 0x7f0428142690 addr v1:172.21.15.114:6789/0 2024-03-29T11:28:13.547 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.555+0000 7f042ded6640 10 monclient(hunting): picked mon.noname-b con 0x7f04280b17d0 addr v1:172.21.15.114:6790/0 2024-03-29T11:28:13.547 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.555+0000 7f042ded6640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0428142c80 con 0x7f0428142690 2024-03-29T11:28:13.548 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.555+0000 7f042ded6640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04280aae40 con 0x7f04281422b0 2024-03-29T11:28:13.548 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.555+0000 7f042ded6640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f042813fbd0 con 0x7f04280b17d0 2024-03-29T11:28:13.548 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.555+0000 7f042ded6640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:13.548 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.555+0000 7f042ded6640 10 monclient(hunting): authenticate will time out at 1363.737466s 2024-03-29T11:28:13.553 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.559+0000 7f0427fff640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f04280b17d0 0x7f04280b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:52897/0 (socket says 172.21.15.114:52897) 2024-03-29T11:28:13.553 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.559+0000 7f0426ffd640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f0428142690 0x7f04280b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:49813/0 (socket says 172.21.15.114:49813) 2024-03-29T11:28:13.553 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.559+0000 7f0427fff640 1 -- 172.21.15.114:0/2582640959 learned_addr learned my addr 172.21.15.114:0/2582640959 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:28:13.554 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.559+0000 7f04267fc640 1 -- 172.21.15.114:0/2582640959 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3276536738 0 0) 0x7f042813fbd0 con 0x7f04280b17d0 2024-03-29T11:28:13.554 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.559+0000 7f04267fc640 10 monclient(hunting): _negotiate 2024-03-29T11:28:13.554 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.559+0000 7f04267fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:13.554 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.559+0000 7f04267fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:13.554 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.559+0000 7f04267fc640 10 monclient(hunting): my global_id is 4514 2024-03-29T11:28:13.554 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.559+0000 7f04267fc640 1 -- 172.21.15.114:0/2582640959 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0410003520 con 0x7f04280b17d0 2024-03-29T11:28:13.554 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.560+0000 7f04267fc640 1 -- 172.21.15.114:0/2582640959 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 669903837 0 0) 0x7f04280aae40 con 0x7f04281422b0 2024-03-29T11:28:13.554 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.560+0000 7f04267fc640 10 monclient(hunting): _negotiate 2024-03-29T11:28:13.554 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.560+0000 7f04267fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:13.554 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.560+0000 7f04267fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:13.554 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.560+0000 7f04267fc640 10 monclient(hunting): my global_id is 4459 2024-03-29T11:28:13.554 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.560+0000 7f04267fc640 1 -- 172.21.15.114:0/2582640959 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f042813fbd0 con 0x7f04281422b0 2024-03-29T11:28:13.554 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.560+0000 7f04267fc640 1 -- 172.21.15.114:0/2582640959 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1375039046 0 0) 0x7f0410003520 con 0x7f04280b17d0 2024-03-29T11:28:13.555 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.560+0000 7f04267fc640 1 -- 172.21.15.114:0/2582640959 >> v1:172.21.15.176:6789/0 conn(0x7f04281422b0 legacy=0x7f042813f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:13.555 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.560+0000 7f04267fc640 1 -- 172.21.15.114:0/2582640959 >> v1:172.21.15.114:6789/0 conn(0x7f0428142690 legacy=0x7f04280b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:13.555 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.560+0000 7f04267fc640 10 monclient: _finish_hunting 0 2024-03-29T11:28:13.555 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.560+0000 7f04267fc640 1 monclient: found mon.noname-b 2024-03-29T11:28:13.555 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.560+0000 7f04267fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:13.555 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.560+0000 7f04267fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.114:6790/0 2024-03-29T11:28:13.555 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.560+0000 7f04267fc640 1 -- 172.21.15.114:0/2582640959 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0428151940 con 0x7f04280b17d0 2024-03-29T11:28:13.555 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.560+0000 7f04267fc640 10 monclient: _finish_auth 0 2024-03-29T11:28:13.555 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.560+0000 7f04267fc640 10 monclient: _check_auth_tickets 2024-03-29T11:28:13.555 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.560+0000 7f04267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:13.555 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.560+0000 7f04267fc640 1 -- 172.21.15.114:0/2582640959 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f0420001a10 con 0x7f04280b17d0 2024-03-29T11:28:13.555 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.560+0000 7f042ded6640 5 monclient: authenticate success, global_id 4514 2024-03-29T11:28:13.555 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.560+0000 7f042ded6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:13.555 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.560+0000 7f04267fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:13.556 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.560+0000 7f04267fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:28:13.556 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.560+0000 7f04267fc640 10 monclient: dump: 2024-03-29T11:28:13.556 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:28:13.556 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:28:13.556 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:28:13.556 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:28:13.556 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:13.556 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:28:13.556 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:28:13.556 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:28:13.556 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:28:13.556 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:28:13.556 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.560+0000 7f042ded6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:13.557 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.561+0000 7f04267fc640 1 -- 172.21.15.114:0/2582640959 <== mon.2 v1:172.21.15.114:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0420001b70 con 0x7f04280b17d0 2024-03-29T11:28:13.557 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.561+0000 7f04267fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:13.557 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.561+0000 7f04267fc640 1 -- 172.21.15.114:0/2582640959 <== mon.2 v1:172.21.15.114:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f0420001f00 con 0x7f04280b17d0 2024-03-29T11:28:13.557 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.561+0000 7f04267fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:13.557 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.561+0000 7f04267fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:28:13.557 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.561+0000 7f04267fc640 10 monclient: dump: 2024-03-29T11:28:13.557 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:28:13.557 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:28:13.557 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:28:13.557 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:28:13.557 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:13.557 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:28:13.557 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:28:13.557 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:28:13.558 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:28:13.558 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:28:13.558 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.561+0000 7f042ded6640 10 monclient: get_monmap_and_config success 2024-03-29T11:28:13.558 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.561+0000 7f042ded6640 4 set_mon_vals no callback set 2024-03-29T11:28:13.558 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.562+0000 7f042ded6640 10 monclient: shutdown 2024-03-29T11:28:13.558 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.562+0000 7f042ded6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:13.558 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.562+0000 7f042ded6640 1 -- 172.21.15.114:0/2582640959 >> v1:172.21.15.114:6790/0 conn(0x7f04280b17d0 legacy=0x7f04280b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:13.558 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.563+0000 7f042ded6640 1 -- 172.21.15.114:0/2582640959 shutdown_connections 2024-03-29T11:28:13.558 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.563+0000 7f042ded6640 1 -- 172.21.15.114:0/2582640959 >> 172.21.15.114:0/2582640959 conn(0x7f0428066a00 msgr2=0x7f0428066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:13.558 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.564+0000 7f042ded6640 1 -- 172.21.15.114:0/2582640959 shutdown_connections 2024-03-29T11:28:13.562 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.564+0000 7f042ded6640 1 -- 172.21.15.114:0/2582640959 wait complete. 2024-03-29T11:28:13.562 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.565+0000 7f042ded6640 10 monclient: build_initial_monmap 2024-03-29T11:28:13.562 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.565+0000 7f042ded6640 10 monclient: monmap: 2024-03-29T11:28:13.562 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:28:13.562 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:28:13.562 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T11:28:13.562 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T11:28:13.562 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:13.562 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:28:13.563 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:28:13.563 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T11:28:13.563 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T11:28:13.563 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:28:13.563 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.566+0000 7f042ded6640 1 librados: starting msgr at 2024-03-29T11:28:13.563 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.566+0000 7f042ded6640 1 librados: starting objecter 2024-03-29T11:28:13.563 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.566+0000 7f042ded6640 1 Processor -- start 2024-03-29T11:28:13.563 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.566+0000 7f042ded6640 1 -- start start 2024-03-29T11:28:13.563 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.566+0000 7f042ded6640 1 librados: setting wanted keys 2024-03-29T11:28:13.563 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.566+0000 7f042ded6640 1 librados: calling monclient init 2024-03-29T11:28:13.563 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.566+0000 7f042ded6640 10 monclient: init 2024-03-29T11:28:13.563 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.567+0000 7f042ded6640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:13.563 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.567+0000 7f042ded6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:28:13.563 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.567+0000 7f042ded6640 10 monclient(hunting): picked mon.noname-b con 0x7f04280b17d0 addr v1:172.21.15.176:6789/0 2024-03-29T11:28:13.564 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.568+0000 7f042ded6640 10 monclient(hunting): picked mon.noname-c con 0x7f04281422b0 addr v1:172.21.15.114:6790/0 2024-03-29T11:28:13.564 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.569+0000 7f042ded6640 10 monclient(hunting): picked mon.noname-a con 0x7f0428142690 addr v1:172.21.15.114:6789/0 2024-03-29T11:28:13.564 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.569+0000 7f042ded6640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0428132510 con 0x7f0428142690 2024-03-29T11:28:13.564 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.569+0000 7f042ded6640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04281321e0 con 0x7f04280b17d0 2024-03-29T11:28:13.564 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.569+0000 7f042ded6640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f042806f2d0 con 0x7f04281422b0 2024-03-29T11:28:13.564 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.569+0000 7f042ded6640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:13.564 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.569+0000 7f042ded6640 10 monclient(hunting): authenticate will time out at 1363.751139s 2024-03-29T11:28:13.564 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.569+0000 7f04277fe640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f04280b17d0 0x7f0428131ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.114:46284/0 (socket says 172.21.15.114:46284) 2024-03-29T11:28:13.564 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.569+0000 7f04277fe640 1 -- 172.21.15.114:0/4214301022 learned_addr learned my addr 172.21.15.114:0/4214301022 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:28:13.564 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.571+0000 7f040ffff640 10 client.?.objecter ms_handle_connect 0x7f04280b17d0 2024-03-29T11:28:13.564 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.571+0000 7f040ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:13.564 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.571+0000 7f040ffff640 1 -- 172.21.15.114:0/4214301022 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2598948130 0 0) 0x7f04281321e0 con 0x7f04280b17d0 2024-03-29T11:28:13.564 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.571+0000 7f040ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:28:13.564 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.571+0000 7f040ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:13.565 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.571+0000 7f040ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:13.565 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.571+0000 7f040ffff640 10 monclient(hunting): my global_id is 4462 2024-03-29T11:28:13.565 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.571+0000 7f040ffff640 1 -- 172.21.15.114:0/4214301022 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0414004eb0 con 0x7f04280b17d0 2024-03-29T11:28:13.565 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.572+0000 7f040ffff640 10 client.?.objecter ms_handle_connect 0x7f04281422b0 2024-03-29T11:28:13.565 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.572+0000 7f040ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:13.565 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.572+0000 7f040ffff640 1 -- 172.21.15.114:0/4214301022 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3866259536 0 0) 0x7f0414004eb0 con 0x7f04280b17d0 2024-03-29T11:28:13.565 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.572+0000 7f040ffff640 1 -- 172.21.15.114:0/4214301022 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f04281321e0 con 0x7f04280b17d0 2024-03-29T11:28:13.565 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.572+0000 7f040ffff640 1 -- 172.21.15.114:0/4214301022 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f0418001a10 con 0x7f04280b17d0 2024-03-29T11:28:13.565 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.572+0000 7f040ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:13.565 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.572+0000 7f040ffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:28:13.565 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.572+0000 7f040ffff640 10 monclient(hunting): dump: 2024-03-29T11:28:13.565 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:28:13.565 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:28:13.565 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:28:13.565 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:28:13.566 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:13.566 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:28:13.566 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:28:13.566 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:28:13.566 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:28:13.566 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:28:13.566 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.572+0000 7f040ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:28:13.566 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.572+0000 7f040ffff640 1 -- 172.21.15.114:0/4214301022 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1808389792 0 0) 0x7f042806f2d0 con 0x7f04281422b0 2024-03-29T11:28:13.566 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.572+0000 7f040ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:28:13.566 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.572+0000 7f040ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:13.566 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.572+0000 7f040ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:13.566 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.572+0000 7f040ffff640 10 monclient(hunting): my global_id is 4517 2024-03-29T11:28:13.566 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.572+0000 7f040ffff640 1 -- 172.21.15.114:0/4214301022 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0414004eb0 con 0x7f04281422b0 2024-03-29T11:28:13.566 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.572+0000 7f040ffff640 1 -- 172.21.15.114:0/4214301022 <== mon.1 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 658507103 0 0) 0x7f04281321e0 con 0x7f04280b17d0 2024-03-29T11:28:13.567 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.572+0000 7f040ffff640 1 -- 172.21.15.114:0/4214301022 >> v1:172.21.15.114:6790/0 conn(0x7f04281422b0 legacy=0x7f04281decc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:13.567 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.572+0000 7f040ffff640 1 -- 172.21.15.114:0/4214301022 >> v1:172.21.15.114:6789/0 conn(0x7f0428142690 legacy=0x7f04281e6200 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:28:13.567 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.573+0000 7f040ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:28:13.567 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.573+0000 7f040ffff640 1 monclient: found mon.b 2024-03-29T11:28:13.567 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.573+0000 7f040ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:13.567 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.573+0000 7f040ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:28:13.567 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.573+0000 7f040ffff640 1 -- 172.21.15.114:0/4214301022 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04281329a0 con 0x7f04280b17d0 2024-03-29T11:28:13.568 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.573+0000 7f040ffff640 10 monclient: _finish_auth 0 2024-03-29T11:28:13.568 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.573+0000 7f040ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:28:13.568 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.573+0000 7f040ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:13.568 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.574+0000 7f042ded6640 5 monclient: authenticate success, global_id 4462 2024-03-29T11:28:13.568 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.574+0000 7f042ded6640 10 monclient: _renew_subs 2024-03-29T11:28:13.568 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.574+0000 7f042ded6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:28:13.568 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.574+0000 7f042ded6640 1 -- 172.21.15.114:0/4214301022 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04281df3d0 con 0x7f04280b17d0 2024-03-29T11:28:13.568 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.575+0000 7f042ded6640 10 client.4462.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:28:13.568 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.575+0000 7f042ded6640 10 monclient: _renew_subs 2024-03-29T11:28:13.568 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.575+0000 7f042ded6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:28:13.568 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.575+0000 7f042ded6640 1 -- 172.21.15.114:0/4214301022 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f042813bcf0 con 0x7f04280b17d0 2024-03-29T11:28:13.568 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.575+0000 7f042ded6640 1 librados: init done 2024-03-29T11:28:13.569 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.579+0000 7f040ffff640 1 -- 172.21.15.114:0/4214301022 <== mon.1 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0418001cd0 con 0x7f04280b17d0 2024-03-29T11:28:13.569 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.579+0000 7f040ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:13.569 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.579+0000 7f040ffff640 1 -- 172.21.15.114:0/4214301022 <== mon.1 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f0418003a60 con 0x7f04280b17d0 2024-03-29T11:28:13.569 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.579+0000 7f040ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:13.569 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.579+0000 7f042ded6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:13.569 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.579+0000 7f040ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:28:13.569 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.579+0000 7f040ffff640 10 monclient: dump: 2024-03-29T11:28:13.569 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:28:13.569 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:28:13.569 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:28:13.569 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:28:13.569 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:13.569 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:28:13.569 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:28:13.570 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:28:13.570 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:28:13.570 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:28:13.570 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.579+0000 7f042ded6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:13.570 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.579+0000 7f042ded6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:28:13.570 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.579+0000 7f042ded6640 1 -- 172.21.15.114:0/4214301022 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0428075f10 con 0x7f04280b17d0 2024-03-29T11:28:13.570 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.579+0000 7f0425ffb640 4 set_mon_vals no callback set 2024-03-29T11:28:13.571 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.581+0000 7f040ffff640 1 -- 172.21.15.114:0/4214301022 <== mon.1 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1594292461 0 0) 0x7f0418003bc0 con 0x7f04280b17d0 2024-03-29T11:28:13.571 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.582+0000 7f040ffff640 1 -- 172.21.15.114:0/4214301022 <== mon.1 v1:172.21.15.176:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3238987313 0 0) 0x7f0418084b70 con 0x7f04280b17d0 2024-03-29T11:28:13.571 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.582+0000 7f040ffff640 10 client.4462.objecter ms_dispatch 0x7f0428000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:28:13.571 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.582+0000 7f040ffff640 3 client.4462.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:28:13.571 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.582+0000 7f040ffff640 3 client.4462.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:28:13.571 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.582+0000 7f040ffff640 20 client.4462.objecter dump_active .. 0 homeless 2024-03-29T11:28:13.572 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.583+0000 7f0426ffd640 10 monclient: get_auth_request con 0x7f041406dd10 auth_method 0 2024-03-29T11:28:13.573 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.583+0000 7f040ffff640 10 client.4462.objecter ms_handle_connect 0x7f041406dd10 2024-03-29T11:28:13.576 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.587+0000 7f040ffff640 1 -- 172.21.15.114:0/4214301022 <== mon.1 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0418050280 con 0x7f04280b17d0 2024-03-29T11:28:13.576 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.587+0000 7f040ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:13.577 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.587+0000 7f040ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:28:13.930 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.940+0000 7f042ded6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:13.931 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.940+0000 7f042ded6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:13.931 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.940+0000 7f042ded6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:28:13.931 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.940+0000 7f042ded6640 1 -- 172.21.15.114:0/4214301022 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f04280b2270 con 0x7f04280b17d0 2024-03-29T11:28:13.931 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.942+0000 7f040ffff640 1 -- 172.21.15.114:0/4214301022 <== mon.1 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2858224613) 0x7f0418052380 con 0x7f04280b17d0 2024-03-29T11:28:13.931 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.942+0000 7f040ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:13.931 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.942+0000 7f040ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:28:13.932 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T11:28:13.932 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"b0de2b9b-c45a-44c7-b6ee-e83b39a1e108","created":"2024-03-29T11:20:49.420660+0000","modified":"2024-03-29T11:20:51.700167+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:51.700167+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a64785d3-6b11-499a-b706-cc827ab96a74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5882dadb-e7a8-4604-ba9a-418013c106c8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d7e8c409-bc66-4e4a-bbdb-420bad31388e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0682989-3595-4cd6-90a5-ab9111c278e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"67068120-e3ff-4817-bd06-44feb7479a35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09e8b-8a85-4997-bdea-75d754e6ee67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2f0d9ead-f47f-4cb3-92a4-0a5ad23cd831","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fdb6a25f-6bd8-4c65-9b9b-b460ba7673a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:28:13.936 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.946+0000 7f042ded6640 10 librados: watch_flush enter 2024-03-29T11:28:13.936 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.947+0000 7f042ded6640 10 librados: watch_flush exit 2024-03-29T11:28:13.937 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.947+0000 7f042ded6640 20 client.4462.objecter shutdown clearing up homeless session... 2024-03-29T11:28:13.937 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.947+0000 7f042ded6640 10 client.4462.objecter successfully canceled tick 2024-03-29T11:28:13.937 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.947+0000 7f042ded6640 1 -- 172.21.15.114:0/4214301022 >> v1:172.21.15.114:6816/4038755055 conn(0x7f041406dd10 legacy=0x7f04140701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:13.937 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.947+0000 7f042ded6640 10 monclient: shutdown 2024-03-29T11:28:13.937 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.947+0000 7f042ded6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:13.937 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.947+0000 7f042ded6640 1 -- 172.21.15.114:0/4214301022 >> v1:172.21.15.176:6789/0 conn(0x7f04280b17d0 legacy=0x7f0428131ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:13.937 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.948+0000 7f042ded6640 1 -- 172.21.15.114:0/4214301022 shutdown_connections 2024-03-29T11:28:13.937 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.948+0000 7f042ded6640 1 -- 172.21.15.114:0/4214301022 >> 172.21.15.114:0/4214301022 conn(0x7f0428066a00 msgr2=0x7f04280b2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:13.938 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.948+0000 7f042ded6640 1 -- 172.21.15.114:0/4214301022 shutdown_connections 2024-03-29T11:28:13.938 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.948+0000 7f042ded6640 1 -- 172.21.15.114:0/4214301022 wait complete. 2024-03-29T11:28:13.938 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:13.949+0000 7f042ded6640 1 librados: shutdown 2024-03-29T11:28:13.957 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:28:20.765 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:28:20.932 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.942+0000 7ff50a5af640 10 monclient: get_monmap_and_config 2024-03-29T11:28:20.932 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.942+0000 7ff50a5af640 10 monclient: build_initial_monmap 2024-03-29T11:28:20.932 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.942+0000 7ff50a5af640 10 monclient: monmap: 2024-03-29T11:28:20.932 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:28:20.932 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:28:20.933 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:28:20.943871+0000 2024-03-29T11:28:20.933 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:28:20.943871+0000 2024-03-29T11:28:20.933 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:20.933 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:28:20.933 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:28:20.933 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T11:28:20.933 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T11:28:20.933 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:28:20.934 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.944+0000 7ff50a5af640 1 Processor -- start 2024-03-29T11:28:20.934 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.944+0000 7ff50a5af640 1 -- start start 2024-03-29T11:28:20.934 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.944+0000 7ff50a5af640 10 monclient: init 2024-03-29T11:28:20.934 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.945+0000 7ff50a5af640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:20.934 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.945+0000 7ff50a5af640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:28:20.934 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.945+0000 7ff50a5af640 10 monclient(hunting): picked mon.noname-c con 0x7ff5041457f0 addr v1:172.21.15.176:6789/0 2024-03-29T11:28:20.934 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.945+0000 7ff50a5af640 10 monclient(hunting): picked mon.noname-a con 0x7ff50413f650 addr v1:172.21.15.114:6789/0 2024-03-29T11:28:20.935 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.945+0000 7ff50a5af640 10 monclient(hunting): picked mon.noname-b con 0x7ff5040a9d70 addr v1:172.21.15.114:6790/0 2024-03-29T11:28:20.935 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.945+0000 7ff50a5af640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5040aaef0 con 0x7ff50413f650 2024-03-29T11:28:20.935 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.945+0000 7ff50a5af640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff504145bd0 con 0x7ff5041457f0 2024-03-29T11:28:20.935 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.945+0000 7ff50a5af640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff504145eb0 con 0x7ff5040a9d70 2024-03-29T11:28:20.935 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.945+0000 7ff50a5af640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:20.935 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.945+0000 7ff50a5af640 10 monclient(hunting): authenticate will time out at 1371.127306s 2024-03-29T11:28:20.936 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.945+0000 7ff503fff640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7ff5041457f0 0x7ff50413ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.114:36038/0 (socket says 172.21.15.114:36038) 2024-03-29T11:28:20.936 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.945+0000 7ff503fff640 1 -- 172.21.15.114:0/1510741373 learned_addr learned my addr 172.21.15.114:0/1510741373 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:28:20.936 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.945+0000 7ff502ffd640 1 -- 172.21.15.114:0/1510741373 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4152345805 0 0) 0x7ff504145bd0 con 0x7ff5041457f0 2024-03-29T11:28:20.936 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.945+0000 7ff502ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:28:20.936 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.945+0000 7ff502ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:20.937 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.945+0000 7ff502ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:20.937 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.945+0000 7ff502ffd640 10 monclient(hunting): my global_id is 4465 2024-03-29T11:28:20.937 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.945+0000 7ff502ffd640 1 -- 172.21.15.114:0/1510741373 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4ec003520 con 0x7ff5041457f0 2024-03-29T11:28:20.937 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.946+0000 7ff502ffd640 1 -- 172.21.15.114:0/1510741373 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 122194704 0 0) 0x7ff4ec003520 con 0x7ff5041457f0 2024-03-29T11:28:20.937 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.946+0000 7ff502ffd640 1 -- 172.21.15.114:0/1510741373 >> v1:172.21.15.114:6790/0 conn(0x7ff5040a9d70 legacy=0x7ff50414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:20.937 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.946+0000 7ff502ffd640 1 -- 172.21.15.114:0/1510741373 >> v1:172.21.15.114:6789/0 conn(0x7ff50413f650 legacy=0x7ff5040b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:20.937 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.946+0000 7ff502ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:28:20.937 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.946+0000 7ff502ffd640 1 monclient: found mon.noname-c 2024-03-29T11:28:20.937 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.946+0000 7ff502ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:20.937 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.946+0000 7ff502ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.176:6789/0 2024-03-29T11:28:20.937 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.946+0000 7ff502ffd640 1 -- 172.21.15.114:0/1510741373 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff504146340 con 0x7ff5041457f0 2024-03-29T11:28:20.937 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.946+0000 7ff502ffd640 10 monclient: _finish_auth 0 2024-03-29T11:28:20.937 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.946+0000 7ff502ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:28:20.937 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.946+0000 7ff502ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:20.938 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.946+0000 7ff502ffd640 1 -- 172.21.15.114:0/1510741373 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7ff4f4001690 con 0x7ff5041457f0 2024-03-29T11:28:20.938 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.946+0000 7ff502ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:20.938 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.946+0000 7ff502ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:28:20.938 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.946+0000 7ff502ffd640 10 monclient: dump: 2024-03-29T11:28:20.938 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:28:20.938 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:28:20.938 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:28:20.938 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:28:20.938 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:20.938 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:28:20.938 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:28:20.938 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:28:20.938 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:28:20.938 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:28:20.939 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.946+0000 7ff50a5af640 5 monclient: authenticate success, global_id 4465 2024-03-29T11:28:20.939 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.946+0000 7ff50a5af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:20.939 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.946+0000 7ff502ffd640 1 -- 172.21.15.114:0/1510741373 <== mon.1 v1:172.21.15.176:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff4f4001860 con 0x7ff5041457f0 2024-03-29T11:28:20.939 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.946+0000 7ff502ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:20.939 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.946+0000 7ff502ffd640 1 -- 172.21.15.114:0/1510741373 <== mon.1 v1:172.21.15.176:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7ff4f4001cb0 con 0x7ff5041457f0 2024-03-29T11:28:20.939 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.946+0000 7ff502ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:20.939 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.946+0000 7ff502ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:28:20.939 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.946+0000 7ff502ffd640 10 monclient: dump: 2024-03-29T11:28:20.939 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:28:20.939 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:28:20.939 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:28:20.939 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:28:20.939 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:20.939 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:28:20.939 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:28:20.940 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:28:20.940 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:28:20.940 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:28:20.940 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.946+0000 7ff50a5af640 10 monclient: get_monmap_and_config success 2024-03-29T11:28:20.940 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.946+0000 7ff50a5af640 4 set_mon_vals no callback set 2024-03-29T11:28:20.942 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.947+0000 7ff50a5af640 10 monclient: shutdown 2024-03-29T11:28:20.942 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.947+0000 7ff50a5af640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:20.943 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.947+0000 7ff50a5af640 1 -- 172.21.15.114:0/1510741373 >> v1:172.21.15.176:6789/0 conn(0x7ff5041457f0 legacy=0x7ff50413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:20.943 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.947+0000 7ff50a5af640 1 -- 172.21.15.114:0/1510741373 shutdown_connections 2024-03-29T11:28:20.943 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.948+0000 7ff50a5af640 1 -- 172.21.15.114:0/1510741373 >> 172.21.15.114:0/1510741373 conn(0x7ff504066a00 msgr2=0x7ff504066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:20.943 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.948+0000 7ff50a5af640 1 -- 172.21.15.114:0/1510741373 shutdown_connections 2024-03-29T11:28:20.943 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.948+0000 7ff50a5af640 1 -- 172.21.15.114:0/1510741373 wait complete. 2024-03-29T11:28:20.943 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.949+0000 7ff50a5af640 10 monclient: build_initial_monmap 2024-03-29T11:28:20.943 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.949+0000 7ff50a5af640 10 monclient: monmap: 2024-03-29T11:28:20.943 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:28:20.943 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:28:20.943 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T11:28:20.943 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T11:28:20.943 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:20.943 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:28:20.943 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:28:20.944 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T11:28:20.944 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T11:28:20.944 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:28:20.944 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.949+0000 7ff50a5af640 1 librados: starting msgr at 2024-03-29T11:28:20.944 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.949+0000 7ff50a5af640 1 librados: starting objecter 2024-03-29T11:28:20.944 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.949+0000 7ff50a5af640 1 Processor -- start 2024-03-29T11:28:20.944 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.950+0000 7ff50a5af640 1 -- start start 2024-03-29T11:28:20.944 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.950+0000 7ff50a5af640 1 librados: setting wanted keys 2024-03-29T11:28:20.944 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.950+0000 7ff50a5af640 1 librados: calling monclient init 2024-03-29T11:28:20.944 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.950+0000 7ff50a5af640 10 monclient: init 2024-03-29T11:28:20.944 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.950+0000 7ff50a5af640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:20.944 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.950+0000 7ff50a5af640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:28:20.944 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.950+0000 7ff50a5af640 10 monclient(hunting): picked mon.noname-b con 0x7ff5040a9d70 addr v1:172.21.15.176:6789/0 2024-03-29T11:28:20.944 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.950+0000 7ff50a5af640 10 monclient(hunting): picked mon.noname-a con 0x7ff50413f650 addr v1:172.21.15.114:6789/0 2024-03-29T11:28:20.945 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.951+0000 7ff50a5af640 10 monclient(hunting): picked mon.noname-c con 0x7ff5041457f0 addr v1:172.21.15.114:6790/0 2024-03-29T11:28:20.945 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.951+0000 7ff50a5af640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff504140720 con 0x7ff50413f650 2024-03-29T11:28:20.945 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.951+0000 7ff50a5af640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5041359f0 con 0x7ff5040a9d70 2024-03-29T11:28:20.945 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.951+0000 7ff50a5af640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5041d7e00 con 0x7ff5041457f0 2024-03-29T11:28:20.945 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.951+0000 7ff50a5af640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:20.945 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.951+0000 7ff50a5af640 10 monclient(hunting): authenticate will time out at 1371.133158s 2024-03-29T11:28:20.945 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.951+0000 7ff508b4c640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7ff5041457f0 0x7ff5041d76f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:42794/0 (socket says 172.21.15.114:42794) 2024-03-29T11:28:20.945 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.951+0000 7ff508b4c640 1 -- 172.21.15.114:0/832839757 learned_addr learned my addr 172.21.15.114:0/832839757 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:28:20.945 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.951+0000 7ff500ff9640 10 client.?.objecter ms_handle_connect 0x7ff5041457f0 2024-03-29T11:28:20.945 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.951+0000 7ff500ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:20.945 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.951+0000 7ff500ff9640 10 client.?.objecter ms_handle_connect 0x7ff50413f650 2024-03-29T11:28:20.945 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.951+0000 7ff500ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:20.945 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.951+0000 7ff500ff9640 10 client.?.objecter ms_handle_connect 0x7ff5040a9d70 2024-03-29T11:28:20.945 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.951+0000 7ff500ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:20.945 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.952+0000 7ff500ff9640 1 -- 172.21.15.114:0/832839757 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3085936038 0 0) 0x7ff5041d7e00 con 0x7ff5041457f0 2024-03-29T11:28:20.946 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.952+0000 7ff500ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:28:20.946 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.952+0000 7ff500ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:20.946 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.952+0000 7ff500ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:20.946 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.952+0000 7ff500ff9640 10 monclient(hunting): my global_id is 4520 2024-03-29T11:28:20.946 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.952+0000 7ff500ff9640 1 -- 172.21.15.114:0/832839757 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4f0004eb0 con 0x7ff5041457f0 2024-03-29T11:28:20.946 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.952+0000 7ff500ff9640 1 -- 172.21.15.114:0/832839757 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 723584187 0 0) 0x7ff504146340 con 0x7ff5040a9d70 2024-03-29T11:28:20.946 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.952+0000 7ff500ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:28:20.946 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.952+0000 7ff500ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:20.946 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.952+0000 7ff500ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:20.946 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.952+0000 7ff500ff9640 10 monclient(hunting): my global_id is 4468 2024-03-29T11:28:20.946 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.952+0000 7ff500ff9640 1 -- 172.21.15.114:0/832839757 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5041d7e00 con 0x7ff5040a9d70 2024-03-29T11:28:20.946 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.952+0000 7ff500ff9640 1 -- 172.21.15.114:0/832839757 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3493369750 0 0) 0x7ff504140720 con 0x7ff50413f650 2024-03-29T11:28:20.946 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.952+0000 7ff500ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:28:20.946 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.952+0000 7ff500ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:20.947 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.952+0000 7ff500ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:20.947 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.952+0000 7ff500ff9640 10 monclient(hunting): my global_id is 4494 2024-03-29T11:28:20.947 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.952+0000 7ff500ff9640 1 -- 172.21.15.114:0/832839757 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff504146340 con 0x7ff50413f650 2024-03-29T11:28:20.947 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.952+0000 7ff500ff9640 1 -- 172.21.15.114:0/832839757 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2553433009 0 0) 0x7ff5041d7e00 con 0x7ff5040a9d70 2024-03-29T11:28:20.947 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.952+0000 7ff500ff9640 1 -- 172.21.15.114:0/832839757 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff504140720 con 0x7ff5040a9d70 2024-03-29T11:28:20.947 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.952+0000 7ff500ff9640 1 -- 172.21.15.114:0/832839757 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7ff4f40019d0 con 0x7ff5040a9d70 2024-03-29T11:28:20.947 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.952+0000 7ff500ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:20.947 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.952+0000 7ff500ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:28:20.947 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.952+0000 7ff500ff9640 10 monclient(hunting): dump: 2024-03-29T11:28:20.947 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:28:20.947 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:28:20.947 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:28:20.947 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:28:20.947 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:20.947 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:28:20.948 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:28:20.948 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:28:20.948 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:28:20.948 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:28:20.948 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.952+0000 7ff500ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:28:20.948 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.953+0000 7ff500ff9640 1 -- 172.21.15.114:0/832839757 <== mon.1 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1645998697 0 0) 0x7ff504140720 con 0x7ff5040a9d70 2024-03-29T11:28:20.948 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.953+0000 7ff500ff9640 1 -- 172.21.15.114:0/832839757 >> v1:172.21.15.114:6790/0 conn(0x7ff5041457f0 legacy=0x7ff5041d76f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:20.948 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.953+0000 7ff500ff9640 1 -- 172.21.15.114:0/832839757 >> v1:172.21.15.114:6789/0 conn(0x7ff50413f650 legacy=0x7ff504140010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:20.948 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.953+0000 7ff500ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:28:20.948 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.953+0000 7ff500ff9640 1 monclient: found mon.b 2024-03-29T11:28:20.948 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.953+0000 7ff500ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:20.948 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.953+0000 7ff500ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:28:20.948 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.953+0000 7ff500ff9640 1 -- 172.21.15.114:0/832839757 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff504140ae0 con 0x7ff5040a9d70 2024-03-29T11:28:20.948 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.953+0000 7ff500ff9640 10 monclient: _finish_auth 0 2024-03-29T11:28:20.949 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.953+0000 7ff500ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:28:20.949 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.953+0000 7ff500ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:20.949 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.953+0000 7ff50a5af640 5 monclient: authenticate success, global_id 4468 2024-03-29T11:28:20.950 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.953+0000 7ff50a5af640 10 monclient: _renew_subs 2024-03-29T11:28:20.950 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.953+0000 7ff50a5af640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:28:20.950 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.953+0000 7ff50a5af640 1 -- 172.21.15.114:0/832839757 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff504141000 con 0x7ff5040a9d70 2024-03-29T11:28:20.950 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.953+0000 7ff50a5af640 10 client.4468.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:28:20.950 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.953+0000 7ff50a5af640 10 monclient: _renew_subs 2024-03-29T11:28:20.950 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.953+0000 7ff50a5af640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:28:20.950 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.953+0000 7ff50a5af640 1 -- 172.21.15.114:0/832839757 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5040b0880 con 0x7ff5040a9d70 2024-03-29T11:28:20.950 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.953+0000 7ff50a5af640 1 librados: init done 2024-03-29T11:28:20.951 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.953+0000 7ff500ff9640 1 -- 172.21.15.114:0/832839757 <== mon.1 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff4f40047e0 con 0x7ff5040a9d70 2024-03-29T11:28:20.951 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.953+0000 7ff500ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:20.951 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.953+0000 7ff500ff9640 1 -- 172.21.15.114:0/832839757 <== mon.1 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7ff4f4004af0 con 0x7ff5040a9d70 2024-03-29T11:28:20.951 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.953+0000 7ff500ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:20.951 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.953+0000 7ff500ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:28:20.952 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.953+0000 7ff5027fc640 4 set_mon_vals no callback set 2024-03-29T11:28:20.952 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.953+0000 7ff500ff9640 10 monclient: dump: 2024-03-29T11:28:20.952 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:28:20.952 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:28:20.952 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:28:20.952 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:28:20.952 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:20.952 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:28:20.952 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:28:20.952 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:28:20.952 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:28:20.952 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:28:20.952 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.956+0000 7ff500ff9640 1 -- 172.21.15.114:0/832839757 <== mon.1 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1594292461 0 0) 0x7ff4f401aac0 con 0x7ff5040a9d70 2024-03-29T11:28:20.952 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.956+0000 7ff50a5af640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:20.952 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.956+0000 7ff50a5af640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:20.953 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.956+0000 7ff50a5af640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:28:20.953 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.956+0000 7ff50a5af640 1 -- 172.21.15.114:0/832839757 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4e4005080 con 0x7ff5040a9d70 2024-03-29T11:28:20.953 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.956+0000 7ff5037fe640 10 monclient: get_auth_request con 0x7ff4f006dea0 auth_method 0 2024-03-29T11:28:20.953 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.956+0000 7ff500ff9640 1 -- 172.21.15.114:0/832839757 <== mon.1 v1:172.21.15.176:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3238987313 0 0) 0x7ff4f4084820 con 0x7ff5040a9d70 2024-03-29T11:28:20.953 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.956+0000 7ff500ff9640 10 client.4468.objecter ms_dispatch 0x7ff504000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:28:20.953 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.956+0000 7ff500ff9640 3 client.4468.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:28:20.953 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.956+0000 7ff500ff9640 3 client.4468.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:28:20.953 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.956+0000 7ff500ff9640 20 client.4468.objecter dump_active .. 0 homeless 2024-03-29T11:28:20.953 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.957+0000 7ff500ff9640 10 client.4468.objecter ms_handle_connect 0x7ff4f006dea0 2024-03-29T11:28:20.953 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.964+0000 7ff500ff9640 1 -- 172.21.15.114:0/832839757 <== mon.1 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff4f4050100 con 0x7ff5040a9d70 2024-03-29T11:28:20.954 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.964+0000 7ff500ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:20.954 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:20.964+0000 7ff500ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:28:21.310 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:21.320+0000 7ff50a5af640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:21.311 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:21.320+0000 7ff50a5af640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:21.311 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:21.320+0000 7ff50a5af640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:28:21.311 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:21.320+0000 7ff50a5af640 1 -- 172.21.15.114:0/832839757 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff4e4005330 con 0x7ff5040a9d70 2024-03-29T11:28:21.311 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:21.321+0000 7ff500ff9640 1 -- 172.21.15.114:0/832839757 <== mon.1 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2858224613) 0x7ff4f4052030 con 0x7ff5040a9d70 2024-03-29T11:28:21.311 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:21.321+0000 7ff500ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:21.311 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:21.321+0000 7ff500ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:28:21.311 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T11:28:21.312 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"b0de2b9b-c45a-44c7-b6ee-e83b39a1e108","created":"2024-03-29T11:20:49.420660+0000","modified":"2024-03-29T11:20:51.700167+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:51.700167+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a64785d3-6b11-499a-b706-cc827ab96a74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5882dadb-e7a8-4604-ba9a-418013c106c8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d7e8c409-bc66-4e4a-bbdb-420bad31388e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0682989-3595-4cd6-90a5-ab9111c278e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"67068120-e3ff-4817-bd06-44feb7479a35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09e8b-8a85-4997-bdea-75d754e6ee67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2f0d9ead-f47f-4cb3-92a4-0a5ad23cd831","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fdb6a25f-6bd8-4c65-9b9b-b460ba7673a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:28:21.316 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:21.325+0000 7ff50a5af640 10 librados: watch_flush enter 2024-03-29T11:28:21.316 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:21.325+0000 7ff50a5af640 10 librados: watch_flush exit 2024-03-29T11:28:21.316 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:21.325+0000 7ff50a5af640 20 client.4468.objecter shutdown clearing up homeless session... 2024-03-29T11:28:21.316 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:21.325+0000 7ff50a5af640 10 client.4468.objecter successfully canceled tick 2024-03-29T11:28:21.316 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:21.325+0000 7ff50a5af640 1 -- 172.21.15.114:0/832839757 >> v1:172.21.15.114:6816/4038755055 conn(0x7ff4f006dea0 legacy=0x7ff4f0070360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:21.316 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:21.325+0000 7ff50a5af640 10 monclient: shutdown 2024-03-29T11:28:21.316 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:21.325+0000 7ff50a5af640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:21.316 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:21.325+0000 7ff50a5af640 1 -- 172.21.15.114:0/832839757 >> v1:172.21.15.176:6789/0 conn(0x7ff5040a9d70 legacy=0x7ff5041428a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:21.316 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:21.326+0000 7ff50a5af640 1 -- 172.21.15.114:0/832839757 shutdown_connections 2024-03-29T11:28:21.316 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:21.326+0000 7ff50a5af640 1 -- 172.21.15.114:0/832839757 >> 172.21.15.114:0/832839757 conn(0x7ff504066a00 msgr2=0x7ff5040b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:21.316 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:21.326+0000 7ff50a5af640 1 -- 172.21.15.114:0/832839757 shutdown_connections 2024-03-29T11:28:21.317 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:21.326+0000 7ff50a5af640 1 -- 172.21.15.114:0/832839757 wait complete. 2024-03-29T11:28:21.317 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:21.326+0000 7ff50a5af640 1 librados: shutdown 2024-03-29T11:28:21.332 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:28:28.140 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:28:28.287 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.297+0000 7fe73f3c8640 10 monclient: get_monmap_and_config 2024-03-29T11:28:28.287 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.297+0000 7fe73f3c8640 10 monclient: build_initial_monmap 2024-03-29T11:28:28.287 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.297+0000 7fe73f3c8640 10 monclient: monmap: 2024-03-29T11:28:28.287 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:28:28.287 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:28:28.287 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:28:28.298538+0000 2024-03-29T11:28:28.288 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:28:28.298538+0000 2024-03-29T11:28:28.288 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:28.288 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:28:28.288 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:28:28.288 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T11:28:28.288 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T11:28:28.288 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:28:28.288 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.298+0000 7fe73f3c8640 1 Processor -- start 2024-03-29T11:28:28.289 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.298+0000 7fe73f3c8640 1 -- start start 2024-03-29T11:28:28.289 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.298+0000 7fe73f3c8640 10 monclient: init 2024-03-29T11:28:28.290 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.299+0000 7fe73f3c8640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:28.290 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.299+0000 7fe73f3c8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:28:28.290 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.299+0000 7fe73f3c8640 10 monclient(hunting): picked mon.noname-c con 0x7fe7381422b0 addr v1:172.21.15.176:6789/0 2024-03-29T11:28:28.290 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.299+0000 7fe73f3c8640 10 monclient(hunting): picked mon.noname-b con 0x7fe738142690 addr v1:172.21.15.114:6790/0 2024-03-29T11:28:28.290 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.299+0000 7fe73f3c8640 10 monclient(hunting): picked mon.noname-a con 0x7fe7380b17d0 addr v1:172.21.15.114:6789/0 2024-03-29T11:28:28.290 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.299+0000 7fe73f3c8640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe738142c80 con 0x7fe7380b17d0 2024-03-29T11:28:28.290 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.299+0000 7fe73f3c8640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7380aae40 con 0x7fe7381422b0 2024-03-29T11:28:28.290 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.299+0000 7fe73f3c8640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe73813fbd0 con 0x7fe738142690 2024-03-29T11:28:28.290 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.299+0000 7fe73f3c8640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:28.290 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.299+0000 7fe73f3c8640 10 monclient(hunting): authenticate will time out at 1378.481846s 2024-03-29T11:28:28.291 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.299+0000 7fe73d965640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fe7380b17d0 0x7fe7380b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:57874/0 (socket says 172.21.15.114:57874) 2024-03-29T11:28:28.291 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.300+0000 7fe73d965640 1 -- 172.21.15.114:0/1832093406 learned_addr learned my addr 172.21.15.114:0/1832093406 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:28:28.294 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.300+0000 7fe72ffff640 1 -- 172.21.15.114:0/1832093406 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 33504792 0 0) 0x7fe73813fbd0 con 0x7fe738142690 2024-03-29T11:28:28.294 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.300+0000 7fe72ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:28:28.294 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.300+0000 7fe72ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:28.294 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.300+0000 7fe72ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:28.294 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.300+0000 7fe72ffff640 10 monclient(hunting): my global_id is 4523 2024-03-29T11:28:28.294 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.300+0000 7fe72ffff640 1 -- 172.21.15.114:0/1832093406 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe724003520 con 0x7fe738142690 2024-03-29T11:28:28.294 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.301+0000 7fe72ffff640 1 -- 172.21.15.114:0/1832093406 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1103945109 0 0) 0x7fe7380aae40 con 0x7fe7381422b0 2024-03-29T11:28:28.294 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.301+0000 7fe72ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:28:28.294 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.301+0000 7fe72ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:28.294 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.301+0000 7fe72ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:28.294 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.301+0000 7fe72ffff640 10 monclient(hunting): my global_id is 4471 2024-03-29T11:28:28.294 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.301+0000 7fe72ffff640 1 -- 172.21.15.114:0/1832093406 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe73813fbd0 con 0x7fe7381422b0 2024-03-29T11:28:28.295 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.301+0000 7fe72ffff640 1 -- 172.21.15.114:0/1832093406 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 977237337 0 0) 0x7fe738142c80 con 0x7fe7380b17d0 2024-03-29T11:28:28.295 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.301+0000 7fe72ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:28:28.295 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.301+0000 7fe72ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:28.295 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.301+0000 7fe72ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:28.295 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.301+0000 7fe72ffff640 10 monclient(hunting): my global_id is 4497 2024-03-29T11:28:28.295 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.301+0000 7fe72ffff640 1 -- 172.21.15.114:0/1832093406 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7380aae40 con 0x7fe7380b17d0 2024-03-29T11:28:28.295 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.301+0000 7fe72ffff640 1 -- 172.21.15.114:0/1832093406 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 807002443 0 0) 0x7fe7380aae40 con 0x7fe7380b17d0 2024-03-29T11:28:28.295 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.301+0000 7fe72ffff640 1 -- 172.21.15.114:0/1832093406 >> v1:172.21.15.114:6790/0 conn(0x7fe738142690 legacy=0x7fe7380b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:28.295 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.301+0000 7fe72ffff640 1 -- 172.21.15.114:0/1832093406 >> v1:172.21.15.176:6789/0 conn(0x7fe7381422b0 legacy=0x7fe73813f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:28.295 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.301+0000 7fe72ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:28:28.295 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.301+0000 7fe72ffff640 1 monclient: found mon.noname-a 2024-03-29T11:28:28.295 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.301+0000 7fe72ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:28.295 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.301+0000 7fe72ffff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.114:6789/0 2024-03-29T11:28:28.295 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.301+0000 7fe72ffff640 1 -- 172.21.15.114:0/1832093406 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe738151940 con 0x7fe7380b17d0 2024-03-29T11:28:28.296 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.301+0000 7fe72ffff640 10 monclient: _finish_auth 0 2024-03-29T11:28:28.296 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.301+0000 7fe72ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:28:28.296 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.301+0000 7fe72ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:28.296 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.301+0000 7fe73f3c8640 5 monclient: authenticate success, global_id 4497 2024-03-29T11:28:28.296 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.301+0000 7fe73f3c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:28.296 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.301+0000 7fe72ffff640 1 -- 172.21.15.114:0/1832093406 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fe7300015b0 con 0x7fe7380b17d0 2024-03-29T11:28:28.296 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.301+0000 7fe72ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:28.296 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.301+0000 7fe72ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:28:28.296 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.301+0000 7fe72ffff640 10 monclient: dump: 2024-03-29T11:28:28.296 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:28:28.296 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:28:28.296 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:28:28.296 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:28:28.296 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:28.296 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:28:28.297 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:28:28.297 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:28:28.297 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:28:28.297 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:28:28.297 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.301+0000 7fe73f3c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:28.297 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.302+0000 7fe72ffff640 1 -- 172.21.15.114:0/1832093406 <== mon.0 v1:172.21.15.114:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe730001790 con 0x7fe7380b17d0 2024-03-29T11:28:28.297 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.302+0000 7fe72ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:28.297 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.302+0000 7fe72ffff640 1 -- 172.21.15.114:0/1832093406 <== mon.0 v1:172.21.15.114:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fe7300015b0 con 0x7fe7380b17d0 2024-03-29T11:28:28.297 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.302+0000 7fe72ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:28.297 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.302+0000 7fe72ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:28:28.297 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.302+0000 7fe72ffff640 10 monclient: dump: 2024-03-29T11:28:28.297 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:28:28.297 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:28:28.297 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:28:28.297 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:28:28.298 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:28.298 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:28:28.298 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:28:28.298 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:28:28.298 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:28:28.298 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:28:28.298 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.302+0000 7fe73f3c8640 10 monclient: get_monmap_and_config success 2024-03-29T11:28:28.298 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.302+0000 7fe73f3c8640 4 set_mon_vals no callback set 2024-03-29T11:28:28.298 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.303+0000 7fe73f3c8640 10 monclient: shutdown 2024-03-29T11:28:28.298 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.303+0000 7fe73f3c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:28.298 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.303+0000 7fe73f3c8640 1 -- 172.21.15.114:0/1832093406 >> v1:172.21.15.114:6789/0 conn(0x7fe7380b17d0 legacy=0x7fe7380b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:28.298 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.303+0000 7fe73f3c8640 1 -- 172.21.15.114:0/1832093406 shutdown_connections 2024-03-29T11:28:28.299 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.303+0000 7fe73f3c8640 1 -- 172.21.15.114:0/1832093406 >> 172.21.15.114:0/1832093406 conn(0x7fe738066a00 msgr2=0x7fe738066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:28.299 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.303+0000 7fe73f3c8640 1 -- 172.21.15.114:0/1832093406 shutdown_connections 2024-03-29T11:28:28.299 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.303+0000 7fe73f3c8640 1 -- 172.21.15.114:0/1832093406 wait complete. 2024-03-29T11:28:28.305 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.304+0000 7fe73f3c8640 10 monclient: build_initial_monmap 2024-03-29T11:28:28.305 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.304+0000 7fe73f3c8640 10 monclient: monmap: 2024-03-29T11:28:28.305 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:28:28.305 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:28:28.305 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T11:28:28.305 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T11:28:28.305 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:28.305 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:28:28.305 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:28:28.305 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T11:28:28.305 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T11:28:28.305 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:28:28.305 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.305+0000 7fe73f3c8640 1 librados: starting msgr at 2024-03-29T11:28:28.306 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.305+0000 7fe73f3c8640 1 librados: starting objecter 2024-03-29T11:28:28.306 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.305+0000 7fe73f3c8640 1 Processor -- start 2024-03-29T11:28:28.306 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.305+0000 7fe73f3c8640 1 -- start start 2024-03-29T11:28:28.306 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.305+0000 7fe73f3c8640 1 librados: setting wanted keys 2024-03-29T11:28:28.306 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.305+0000 7fe73f3c8640 1 librados: calling monclient init 2024-03-29T11:28:28.306 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.305+0000 7fe73f3c8640 10 monclient: init 2024-03-29T11:28:28.306 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.306+0000 7fe73f3c8640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:28.306 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.306+0000 7fe73f3c8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:28:28.306 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.307+0000 7fe73f3c8640 10 monclient(hunting): picked mon.noname-b con 0x7fe7380b17d0 addr v1:172.21.15.176:6789/0 2024-03-29T11:28:28.306 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.307+0000 7fe73f3c8640 10 monclient(hunting): picked mon.noname-c con 0x7fe7381422b0 addr v1:172.21.15.114:6790/0 2024-03-29T11:28:28.306 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.307+0000 7fe73f3c8640 10 monclient(hunting): picked mon.noname-a con 0x7fe738142690 addr v1:172.21.15.114:6789/0 2024-03-29T11:28:28.306 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.307+0000 7fe73f3c8640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7380b03d0 con 0x7fe738142690 2024-03-29T11:28:28.306 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.307+0000 7fe73f3c8640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7380af6e0 con 0x7fe7380b17d0 2024-03-29T11:28:28.306 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.307+0000 7fe73f3c8640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7380b0b90 con 0x7fe7381422b0 2024-03-29T11:28:28.307 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.307+0000 7fe73f3c8640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:28.307 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.307+0000 7fe73f3c8640 10 monclient(hunting): authenticate will time out at 1378.489221s 2024-03-29T11:28:28.307 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.307+0000 7fe73d164640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7fe7380b17d0 0x7fe7380aefd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.114:56984/0 (socket says 172.21.15.114:56984) 2024-03-29T11:28:28.307 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.307+0000 7fe73d164640 1 -- 172.21.15.114:0/3910170832 learned_addr learned my addr 172.21.15.114:0/3910170832 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:28:28.307 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.307+0000 7fe72dffb640 10 client.?.objecter ms_handle_connect 0x7fe7380b17d0 2024-03-29T11:28:28.307 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.307+0000 7fe72dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:28.307 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.307+0000 7fe72dffb640 10 client.?.objecter ms_handle_connect 0x7fe7381422b0 2024-03-29T11:28:28.307 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.307+0000 7fe72dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:28.307 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.307+0000 7fe72dffb640 1 -- 172.21.15.114:0/3910170832 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1991532124 0 0) 0x7fe7380af6e0 con 0x7fe7380b17d0 2024-03-29T11:28:28.307 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.307+0000 7fe72dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:28:28.307 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.307+0000 7fe72dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:28.307 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.307+0000 7fe72dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:28.307 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.307+0000 7fe72dffb640 10 monclient(hunting): my global_id is 4474 2024-03-29T11:28:28.307 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.307+0000 7fe72dffb640 1 -- 172.21.15.114:0/3910170832 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe71c004eb0 con 0x7fe7380b17d0 2024-03-29T11:28:28.308 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.308+0000 7fe72dffb640 1 -- 172.21.15.114:0/3910170832 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3410360541 0 0) 0x7fe7380b0b90 con 0x7fe7381422b0 2024-03-29T11:28:28.308 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.308+0000 7fe72dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:28:28.308 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.308+0000 7fe72dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:28.308 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.308+0000 7fe72dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:28.308 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.308+0000 7fe72dffb640 10 monclient(hunting): my global_id is 4526 2024-03-29T11:28:28.308 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.308+0000 7fe72dffb640 1 -- 172.21.15.114:0/3910170832 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7380af6e0 con 0x7fe7381422b0 2024-03-29T11:28:28.308 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.308+0000 7fe72dffb640 10 client.?.objecter ms_handle_connect 0x7fe738142690 2024-03-29T11:28:28.308 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.308+0000 7fe72dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:28.308 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.308+0000 7fe72dffb640 1 -- 172.21.15.114:0/3910170832 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1641603329 0 0) 0x7fe71c004eb0 con 0x7fe7380b17d0 2024-03-29T11:28:28.308 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.308+0000 7fe72dffb640 1 -- 172.21.15.114:0/3910170832 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe7380b0b90 con 0x7fe7380b17d0 2024-03-29T11:28:28.308 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.308+0000 7fe72dffb640 1 -- 172.21.15.114:0/3910170832 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fe734001d10 con 0x7fe7380b17d0 2024-03-29T11:28:28.308 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.308+0000 7fe72dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:28.308 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.308+0000 7fe72dffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:28:28.308 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.308+0000 7fe72dffb640 10 monclient(hunting): dump: 2024-03-29T11:28:28.309 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:28:28.309 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:28:28.309 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:28:28.309 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:28:28.309 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:28.309 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:28:28.309 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:28:28.309 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:28:28.309 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:28:28.309 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:28:28.309 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.308+0000 7fe72dffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:28:28.309 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.308+0000 7fe72dffb640 1 -- 172.21.15.114:0/3910170832 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 668101559 0 0) 0x7fe7380b03d0 con 0x7fe738142690 2024-03-29T11:28:28.309 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.308+0000 7fe72dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:28:28.309 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.308+0000 7fe72dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:28.309 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.308+0000 7fe72dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:28.310 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.308+0000 7fe72dffb640 10 monclient(hunting): my global_id is 4500 2024-03-29T11:28:28.310 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.308+0000 7fe72dffb640 1 -- 172.21.15.114:0/3910170832 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe71c004eb0 con 0x7fe738142690 2024-03-29T11:28:28.310 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.309+0000 7fe72dffb640 1 -- 172.21.15.114:0/3910170832 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 785093318 0 0) 0x7fe7380af6e0 con 0x7fe7381422b0 2024-03-29T11:28:28.310 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.309+0000 7fe72dffb640 1 -- 172.21.15.114:0/3910170832 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe7380b03d0 con 0x7fe7381422b0 2024-03-29T11:28:28.310 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.309+0000 7fe72dffb640 1 -- 172.21.15.114:0/3910170832 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fe7280017b0 con 0x7fe7381422b0 2024-03-29T11:28:28.310 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.309+0000 7fe72dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:28.310 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.309+0000 7fe72dffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:28:28.310 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.309+0000 7fe72dffb640 10 monclient(hunting): dump: 2024-03-29T11:28:28.310 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:28:28.311 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:28:28.311 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:28:28.311 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:28:28.311 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:28.311 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:28:28.311 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:28:28.311 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:28:28.311 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:28:28.312 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:28:28.312 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.309+0000 7fe72dffb640 1 -- 172.21.15.114:0/3910170832 <== mon.1 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1625772938 0 0) 0x7fe7380b0b90 con 0x7fe7380b17d0 2024-03-29T11:28:28.312 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.309+0000 7fe72dffb640 1 -- 172.21.15.114:0/3910170832 >> v1:172.21.15.114:6790/0 conn(0x7fe7381422b0 legacy=0x7fe7381da990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:28.312 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.309+0000 7fe72dffb640 1 -- 172.21.15.114:0/3910170832 >> v1:172.21.15.114:6789/0 conn(0x7fe738142690 legacy=0x7fe7381e1ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:28.312 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.309+0000 7fe72dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:28:28.312 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.309+0000 7fe72dffb640 1 monclient: found mon.b 2024-03-29T11:28:28.312 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.309+0000 7fe72dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:28.312 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.309+0000 7fe72dffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:28:28.312 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.309+0000 7fe72dffb640 1 -- 172.21.15.114:0/3910170832 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7380b0e70 con 0x7fe7380b17d0 2024-03-29T11:28:28.312 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.309+0000 7fe72dffb640 10 monclient: _finish_auth 0 2024-03-29T11:28:28.312 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.309+0000 7fe72dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:28:28.312 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.309+0000 7fe72dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:28.312 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.309+0000 7fe73f3c8640 5 monclient: authenticate success, global_id 4474 2024-03-29T11:28:28.312 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.309+0000 7fe73f3c8640 10 monclient: _renew_subs 2024-03-29T11:28:28.313 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.309+0000 7fe73f3c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:28:28.313 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.309+0000 7fe73f3c8640 1 -- 172.21.15.114:0/3910170832 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7380b1150 con 0x7fe7380b17d0 2024-03-29T11:28:28.313 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.310+0000 7fe73f3c8640 10 client.4474.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:28:28.313 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.310+0000 7fe72dffb640 1 -- 172.21.15.114:0/3910170832 <== mon.1 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe734003990 con 0x7fe7380b17d0 2024-03-29T11:28:28.313 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.310+0000 7fe72dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:28.313 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.310+0000 7fe72dffb640 1 -- 172.21.15.114:0/3910170832 <== mon.1 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fe734003ce0 con 0x7fe7380b17d0 2024-03-29T11:28:28.313 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.310+0000 7fe72dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:28.313 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.310+0000 7fe72dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:28:28.313 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.310+0000 7fe72dffb640 10 monclient: dump: 2024-03-29T11:28:28.313 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:28:28.313 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:28:28.313 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:28:28.313 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:28:28.313 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:28.313 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:28:28.314 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:28:28.314 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:28:28.314 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:28:28.314 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:28:28.314 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.310+0000 7fe73f3c8640 10 monclient: _renew_subs 2024-03-29T11:28:28.314 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.310+0000 7fe73f3c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:28:28.314 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.310+0000 7fe73f3c8640 1 -- 172.21.15.114:0/3910170832 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7381db0a0 con 0x7fe7380b17d0 2024-03-29T11:28:28.314 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.310+0000 7fe72f7fe640 4 set_mon_vals no callback set 2024-03-29T11:28:28.314 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.310+0000 7fe73f3c8640 1 librados: init done 2024-03-29T11:28:28.314 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.311+0000 7fe72dffb640 1 -- 172.21.15.114:0/3910170832 <== mon.1 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1594292461 0 0) 0x7fe734003e40 con 0x7fe7380b17d0 2024-03-29T11:28:28.315 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.312+0000 7fe73c963640 10 monclient: get_auth_request con 0x7fe71c06de20 auth_method 0 2024-03-29T11:28:28.315 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.313+0000 7fe72dffb640 1 -- 172.21.15.114:0/3910170832 <== mon.1 v1:172.21.15.176:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3238987313 0 0) 0x7fe734084dc0 con 0x7fe7380b17d0 2024-03-29T11:28:28.315 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.313+0000 7fe72dffb640 10 client.4474.objecter ms_dispatch 0x7fe738000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:28:28.315 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.313+0000 7fe72dffb640 3 client.4474.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:28:28.315 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.313+0000 7fe72dffb640 3 client.4474.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:28:28.315 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.313+0000 7fe72dffb640 20 client.4474.objecter dump_active .. 0 homeless 2024-03-29T11:28:28.315 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.313+0000 7fe72dffb640 10 client.4474.objecter ms_handle_connect 0x7fe71c06de20 2024-03-29T11:28:28.315 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.313+0000 7fe73f3c8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:28.315 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.313+0000 7fe73f3c8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:28.315 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.313+0000 7fe73f3c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:28:28.315 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.313+0000 7fe73f3c8640 1 -- 172.21.15.114:0/3910170832 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe738075f10 con 0x7fe7380b17d0 2024-03-29T11:28:28.316 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.323+0000 7fe72dffb640 1 -- 172.21.15.114:0/3910170832 <== mon.1 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe7340506a0 con 0x7fe7380b17d0 2024-03-29T11:28:28.316 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.323+0000 7fe72dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:28.316 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.323+0000 7fe72dffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:28:28.684 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.694+0000 7fe73f3c8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:28.684 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.694+0000 7fe73f3c8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:28.684 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.694+0000 7fe73f3c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:28:28.684 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.694+0000 7fe73f3c8640 1 -- 172.21.15.114:0/3910170832 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe7380b2270 con 0x7fe7380b17d0 2024-03-29T11:28:28.686 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.696+0000 7fe72dffb640 1 -- 172.21.15.114:0/3910170832 <== mon.1 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2858224613) 0x7fe7340525d0 con 0x7fe7380b17d0 2024-03-29T11:28:28.686 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.696+0000 7fe72dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:28.686 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.696+0000 7fe72dffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:28:28.687 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T11:28:28.687 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"b0de2b9b-c45a-44c7-b6ee-e83b39a1e108","created":"2024-03-29T11:20:49.420660+0000","modified":"2024-03-29T11:20:51.700167+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:51.700167+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a64785d3-6b11-499a-b706-cc827ab96a74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5882dadb-e7a8-4604-ba9a-418013c106c8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d7e8c409-bc66-4e4a-bbdb-420bad31388e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0682989-3595-4cd6-90a5-ab9111c278e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"67068120-e3ff-4817-bd06-44feb7479a35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09e8b-8a85-4997-bdea-75d754e6ee67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2f0d9ead-f47f-4cb3-92a4-0a5ad23cd831","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fdb6a25f-6bd8-4c65-9b9b-b460ba7673a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:28:28.690 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.700+0000 7fe73f3c8640 10 librados: watch_flush enter 2024-03-29T11:28:28.690 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.700+0000 7fe73f3c8640 10 librados: watch_flush exit 2024-03-29T11:28:28.690 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.700+0000 7fe73f3c8640 20 client.4474.objecter shutdown clearing up homeless session... 2024-03-29T11:28:28.690 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.700+0000 7fe73f3c8640 10 client.4474.objecter successfully canceled tick 2024-03-29T11:28:28.690 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.700+0000 7fe73f3c8640 1 -- 172.21.15.114:0/3910170832 >> v1:172.21.15.114:6816/4038755055 conn(0x7fe71c06de20 legacy=0x7fe71c0702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:28.691 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.700+0000 7fe73f3c8640 10 monclient: shutdown 2024-03-29T11:28:28.691 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.700+0000 7fe73f3c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:28.691 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.700+0000 7fe73f3c8640 1 -- 172.21.15.114:0/3910170832 >> v1:172.21.15.176:6789/0 conn(0x7fe7380b17d0 legacy=0x7fe7380aefd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:28.691 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.700+0000 7fe73f3c8640 1 -- 172.21.15.114:0/3910170832 shutdown_connections 2024-03-29T11:28:28.691 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.700+0000 7fe73f3c8640 1 -- 172.21.15.114:0/3910170832 >> 172.21.15.114:0/3910170832 conn(0x7fe738066a00 msgr2=0x7fe7380b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:28.691 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.700+0000 7fe73f3c8640 1 -- 172.21.15.114:0/3910170832 shutdown_connections 2024-03-29T11:28:28.691 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.701+0000 7fe73f3c8640 1 -- 172.21.15.114:0/3910170832 wait complete. 2024-03-29T11:28:28.691 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:28.701+0000 7fe73f3c8640 1 librados: shutdown 2024-03-29T11:28:28.705 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:28:35.512 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:28:35.649 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.659+0000 7fabd42d3640 10 monclient: get_monmap_and_config 2024-03-29T11:28:35.649 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.659+0000 7fabd42d3640 10 monclient: build_initial_monmap 2024-03-29T11:28:35.649 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.659+0000 7fabd42d3640 10 monclient: monmap: 2024-03-29T11:28:35.649 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:28:35.649 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:28:35.649 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:28:35.660115+0000 2024-03-29T11:28:35.649 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:28:35.660115+0000 2024-03-29T11:28:35.649 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:35.649 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:28:35.649 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:28:35.649 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T11:28:35.649 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T11:28:35.650 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:28:35.650 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.660+0000 7fabd42d3640 1 Processor -- start 2024-03-29T11:28:35.650 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.660+0000 7fabd42d3640 1 -- start start 2024-03-29T11:28:35.650 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.660+0000 7fabd42d3640 10 monclient: init 2024-03-29T11:28:35.653 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.661+0000 7fabd42d3640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:35.653 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.661+0000 7fabd42d3640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:28:35.653 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.661+0000 7fabd42d3640 10 monclient(hunting): picked mon.noname-b con 0x7fabcc1422b0 addr v1:172.21.15.114:6790/0 2024-03-29T11:28:35.653 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.661+0000 7fabd42d3640 10 monclient(hunting): picked mon.noname-a con 0x7fabcc142690 addr v1:172.21.15.114:6789/0 2024-03-29T11:28:35.653 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.661+0000 7fabd42d3640 10 monclient(hunting): picked mon.noname-c con 0x7fabcc0b17d0 addr v1:172.21.15.176:6789/0 2024-03-29T11:28:35.653 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.661+0000 7fabd42d3640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabcc142c80 con 0x7fabcc142690 2024-03-29T11:28:35.653 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.661+0000 7fabd42d3640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabcc0aae40 con 0x7fabcc0b17d0 2024-03-29T11:28:35.653 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.661+0000 7fabd42d3640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabcc13fbd0 con 0x7fabcc1422b0 2024-03-29T11:28:35.653 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.661+0000 7fabd42d3640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:35.653 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.661+0000 7fabd42d3640 10 monclient(hunting): authenticate will time out at 1385.844009s 2024-03-29T11:28:35.653 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.662+0000 7fabd206f640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7fabcc1422b0 0x7fabcc13f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:46450/0 (socket says 172.21.15.114:46450) 2024-03-29T11:28:35.653 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.662+0000 7fabd206f640 1 -- 172.21.15.114:0/2125595065 learned_addr learned my addr 172.21.15.114:0/2125595065 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:28:35.654 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.662+0000 7fabd106d640 1 -- 172.21.15.114:0/2125595065 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3076138557 0 0) 0x7fabcc13fbd0 con 0x7fabcc1422b0 2024-03-29T11:28:35.654 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.662+0000 7fabd106d640 10 monclient(hunting): _negotiate 2024-03-29T11:28:35.654 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.662+0000 7fabd106d640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:35.654 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.662+0000 7fabd106d640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:35.654 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.662+0000 7fabd106d640 10 monclient(hunting): my global_id is 4529 2024-03-29T11:28:35.654 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.662+0000 7fabd106d640 1 -- 172.21.15.114:0/2125595065 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabb8003520 con 0x7fabcc1422b0 2024-03-29T11:28:35.654 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.662+0000 7fabd106d640 1 -- 172.21.15.114:0/2125595065 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 490619470 0 0) 0x7fabcc142c80 con 0x7fabcc142690 2024-03-29T11:28:35.654 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.662+0000 7fabd106d640 10 monclient(hunting): _negotiate 2024-03-29T11:28:35.654 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.662+0000 7fabd106d640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:35.655 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.662+0000 7fabd106d640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:35.655 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.662+0000 7fabd106d640 10 monclient(hunting): my global_id is 4503 2024-03-29T11:28:35.655 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.662+0000 7fabd106d640 1 -- 172.21.15.114:0/2125595065 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabcc13fbd0 con 0x7fabcc142690 2024-03-29T11:28:35.655 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.662+0000 7fabd106d640 1 -- 172.21.15.114:0/2125595065 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2637272883 0 0) 0x7fabb8003520 con 0x7fabcc1422b0 2024-03-29T11:28:35.655 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.662+0000 7fabd106d640 1 -- 172.21.15.114:0/2125595065 >> v1:172.21.15.176:6789/0 conn(0x7fabcc0b17d0 legacy=0x7fabcc0b2ca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:28:35.657 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.665+0000 7fabd106d640 1 -- 172.21.15.114:0/2125595065 >> v1:172.21.15.114:6789/0 conn(0x7fabcc142690 legacy=0x7fabcc0b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:35.657 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.665+0000 7fabd106d640 10 monclient: _finish_hunting 0 2024-03-29T11:28:35.657 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.665+0000 7fabd106d640 1 monclient: found mon.noname-b 2024-03-29T11:28:35.657 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.665+0000 7fabd106d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:35.658 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.665+0000 7fabd106d640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.114:6790/0 2024-03-29T11:28:35.658 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.665+0000 7fabd106d640 1 -- 172.21.15.114:0/2125595065 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabcc151940 con 0x7fabcc1422b0 2024-03-29T11:28:35.658 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.665+0000 7fabd106d640 10 monclient: _finish_auth 0 2024-03-29T11:28:35.658 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.665+0000 7fabd106d640 10 monclient: _check_auth_tickets 2024-03-29T11:28:35.658 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.665+0000 7fabd106d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:35.658 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.666+0000 7fabd42d3640 5 monclient: authenticate success, global_id 4529 2024-03-29T11:28:35.658 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.666+0000 7fabd42d3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:35.658 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.666+0000 7fabd106d640 1 -- 172.21.15.114:0/2125595065 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fabc4001a10 con 0x7fabcc1422b0 2024-03-29T11:28:35.658 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.666+0000 7fabd106d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:35.658 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.666+0000 7fabd106d640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:28:35.658 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.666+0000 7fabd106d640 10 monclient: dump: 2024-03-29T11:28:35.658 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:28:35.658 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:28:35.658 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:28:35.659 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:28:35.659 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:35.659 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:28:35.659 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:28:35.659 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:28:35.659 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:28:35.659 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:28:35.659 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.666+0000 7fabd106d640 1 -- 172.21.15.114:0/2125595065 <== mon.2 v1:172.21.15.114:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fabc4001c30 con 0x7fabcc1422b0 2024-03-29T11:28:35.659 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.666+0000 7fabd106d640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:35.659 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.666+0000 7fabd106d640 1 -- 172.21.15.114:0/2125595065 <== mon.2 v1:172.21.15.114:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fabc4001fc0 con 0x7fabcc1422b0 2024-03-29T11:28:35.659 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.666+0000 7fabd106d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:35.659 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.666+0000 7fabd106d640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:28:35.659 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.666+0000 7fabd106d640 10 monclient: dump: 2024-03-29T11:28:35.659 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:28:35.659 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:28:35.659 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:28:35.660 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:28:35.660 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:35.660 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:28:35.660 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:28:35.660 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:28:35.660 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:28:35.660 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:28:35.660 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.666+0000 7fabd42d3640 10 monclient: get_monmap_and_config success 2024-03-29T11:28:35.660 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.666+0000 7fabd42d3640 4 set_mon_vals no callback set 2024-03-29T11:28:35.660 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.667+0000 7fabd42d3640 10 monclient: shutdown 2024-03-29T11:28:35.660 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.667+0000 7fabd42d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:35.660 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.667+0000 7fabd42d3640 1 -- 172.21.15.114:0/2125595065 >> v1:172.21.15.114:6790/0 conn(0x7fabcc1422b0 legacy=0x7fabcc13f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:35.661 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.669+0000 7fabd42d3640 1 -- 172.21.15.114:0/2125595065 shutdown_connections 2024-03-29T11:28:35.661 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.669+0000 7fabd42d3640 1 -- 172.21.15.114:0/2125595065 >> 172.21.15.114:0/2125595065 conn(0x7fabcc066a00 msgr2=0x7fabcc066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:35.661 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.670+0000 7fabd42d3640 1 -- 172.21.15.114:0/2125595065 shutdown_connections 2024-03-29T11:28:35.661 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.670+0000 7fabd42d3640 1 -- 172.21.15.114:0/2125595065 wait complete. 2024-03-29T11:28:35.662 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.671+0000 7fabd42d3640 10 monclient: build_initial_monmap 2024-03-29T11:28:35.662 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.671+0000 7fabd42d3640 10 monclient: monmap: 2024-03-29T11:28:35.662 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:28:35.662 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:28:35.662 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T11:28:35.662 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T11:28:35.662 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:35.662 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:28:35.662 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:28:35.663 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T11:28:35.663 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T11:28:35.663 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:28:35.663 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.671+0000 7fabd42d3640 1 librados: starting msgr at 2024-03-29T11:28:35.663 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.671+0000 7fabd42d3640 1 librados: starting objecter 2024-03-29T11:28:35.663 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.672+0000 7fabd42d3640 1 Processor -- start 2024-03-29T11:28:35.666 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.672+0000 7fabd42d3640 1 -- start start 2024-03-29T11:28:35.666 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.672+0000 7fabd42d3640 1 librados: setting wanted keys 2024-03-29T11:28:35.667 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.672+0000 7fabd42d3640 1 librados: calling monclient init 2024-03-29T11:28:35.667 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.672+0000 7fabd42d3640 10 monclient: init 2024-03-29T11:28:35.667 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.673+0000 7fabd42d3640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:35.667 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.673+0000 7fabd42d3640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:28:35.667 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.673+0000 7fabd42d3640 10 monclient(hunting): picked mon.noname-a con 0x7fabcc0b17d0 addr v1:172.21.15.114:6789/0 2024-03-29T11:28:35.667 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.673+0000 7fabd42d3640 10 monclient(hunting): picked mon.noname-c con 0x7fabcc1422b0 addr v1:172.21.15.114:6790/0 2024-03-29T11:28:35.667 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.673+0000 7fabd42d3640 10 monclient(hunting): picked mon.noname-b con 0x7fabcc142690 addr v1:172.21.15.176:6789/0 2024-03-29T11:28:35.667 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.673+0000 7fabd42d3640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabcc12dd80 con 0x7fabcc0b17d0 2024-03-29T11:28:35.667 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.673+0000 7fabd42d3640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabcc0ab540 con 0x7fabcc142690 2024-03-29T11:28:35.667 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.673+0000 7fabd42d3640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabcc0ac730 con 0x7fabcc1422b0 2024-03-29T11:28:35.667 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.673+0000 7fabd42d3640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:35.667 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.673+0000 7fabd42d3640 10 monclient(hunting): authenticate will time out at 1385.855355s 2024-03-29T11:28:35.667 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.673+0000 7fabd186e640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7fabcc1422b0 0x7fabcc1304f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:47788/0 (socket says 172.21.15.114:47788) 2024-03-29T11:28:35.667 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.673+0000 7fabd186e640 1 -- 172.21.15.114:0/932624131 learned_addr learned my addr 172.21.15.114:0/932624131 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:28:35.668 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.673+0000 7fabbeffd640 10 client.?.objecter ms_handle_connect 0x7fabcc1422b0 2024-03-29T11:28:35.668 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.673+0000 7fabbeffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:35.668 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.673+0000 7fabbeffd640 1 -- 172.21.15.114:0/932624131 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3358289861 0 0) 0x7fabcc0ac730 con 0x7fabcc1422b0 2024-03-29T11:28:35.668 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.673+0000 7fabbeffd640 10 monclient(hunting): _negotiate 2024-03-29T11:28:35.668 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.673+0000 7fabbeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:35.668 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.673+0000 7fabbeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:35.668 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.673+0000 7fabbeffd640 10 monclient(hunting): my global_id is 4532 2024-03-29T11:28:35.668 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.673+0000 7fabbeffd640 1 -- 172.21.15.114:0/932624131 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabb8003520 con 0x7fabcc1422b0 2024-03-29T11:28:35.668 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.673+0000 7fabbeffd640 1 -- 172.21.15.114:0/932624131 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2838108704 0 0) 0x7fabb8003520 con 0x7fabcc1422b0 2024-03-29T11:28:35.668 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.674+0000 7fabbeffd640 1 -- 172.21.15.114:0/932624131 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fabcc0ac730 con 0x7fabcc1422b0 2024-03-29T11:28:35.668 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.674+0000 7fabbeffd640 1 -- 172.21.15.114:0/932624131 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fabc0001a20 con 0x7fabcc1422b0 2024-03-29T11:28:35.668 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.674+0000 7fabbeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:35.668 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.674+0000 7fabbeffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:28:35.668 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.674+0000 7fabbeffd640 10 monclient(hunting): dump: 2024-03-29T11:28:35.668 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:28:35.669 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:28:35.669 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:28:35.669 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:28:35.669 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:35.669 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:28:35.669 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:28:35.669 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:28:35.669 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:28:35.669 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:28:35.669 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.674+0000 7fabbeffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:28:35.669 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.674+0000 7fabbeffd640 1 -- 172.21.15.114:0/932624131 <== mon.2 v1:172.21.15.114:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3654444722 0 0) 0x7fabcc0ac730 con 0x7fabcc1422b0 2024-03-29T11:28:35.669 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.674+0000 7fabbeffd640 1 -- 172.21.15.114:0/932624131 >> v1:172.21.15.176:6789/0 conn(0x7fabcc142690 legacy=0x7fabcc1e1c80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:28:35.669 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.674+0000 7fabbeffd640 1 -- 172.21.15.114:0/932624131 >> v1:172.21.15.114:6789/0 conn(0x7fabcc0b17d0 legacy=0x7fabcc12fde0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:28:35.669 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.674+0000 7fabbeffd640 10 monclient: _finish_hunting 0 2024-03-29T11:28:35.669 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.674+0000 7fabbeffd640 1 monclient: found mon.c 2024-03-29T11:28:35.670 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.674+0000 7fabbeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:35.670 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.674+0000 7fabbeffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:28:35.670 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.674+0000 7fabbeffd640 1 -- 172.21.15.114:0/932624131 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabcc0aca10 con 0x7fabcc1422b0 2024-03-29T11:28:35.670 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.674+0000 7fabbeffd640 10 monclient: _finish_auth 0 2024-03-29T11:28:35.670 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.674+0000 7fabbeffd640 10 monclient: _check_auth_tickets 2024-03-29T11:28:35.670 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.674+0000 7fabbeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:35.670 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.674+0000 7fabbeffd640 1 -- 172.21.15.114:0/932624131 <== mon.2 v1:172.21.15.114:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fabc0001c80 con 0x7fabcc1422b0 2024-03-29T11:28:35.670 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.674+0000 7fabbeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:35.670 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.674+0000 7fabbffff640 4 set_mon_vals no callback set 2024-03-29T11:28:35.670 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.675+0000 7fabd42d3640 5 monclient: authenticate success, global_id 4532 2024-03-29T11:28:35.670 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.675+0000 7fabd42d3640 10 monclient: _renew_subs 2024-03-29T11:28:35.670 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.675+0000 7fabd42d3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:28:35.670 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.675+0000 7fabd42d3640 1 -- 172.21.15.114:0/932624131 --> v1:172.21.15.114:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabcc0accf0 con 0x7fabcc1422b0 2024-03-29T11:28:35.670 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.675+0000 7fabd42d3640 10 client.4532.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:28:35.670 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.675+0000 7fabd42d3640 10 monclient: _renew_subs 2024-03-29T11:28:35.671 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.675+0000 7fabd42d3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:28:35.671 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.675+0000 7fabd42d3640 1 -- 172.21.15.114:0/932624131 --> v1:172.21.15.114:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fabcc132280 con 0x7fabcc1422b0 2024-03-29T11:28:35.671 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.675+0000 7fabd42d3640 1 librados: init done 2024-03-29T11:28:35.671 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.676+0000 7fabd42d3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:35.671 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.676+0000 7fabbeffd640 1 -- 172.21.15.114:0/932624131 <== mon.2 v1:172.21.15.114:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fabc0003a80 con 0x7fabcc1422b0 2024-03-29T11:28:35.671 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.676+0000 7fabbeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:35.671 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.676+0000 7fabbeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:28:35.671 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.676+0000 7fabbeffd640 10 monclient: dump: 2024-03-29T11:28:35.671 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:28:35.671 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:28:35.671 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:28:35.671 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:28:35.671 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:35.671 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:28:35.671 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:28:35.672 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:28:35.672 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:28:35.672 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:28:35.672 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.676+0000 7fabd42d3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:35.672 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.676+0000 7fabd42d3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:28:35.672 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.676+0000 7fabd42d3640 1 -- 172.21.15.114:0/932624131 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabcc075f10 con 0x7fabcc1422b0 2024-03-29T11:28:35.673 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.678+0000 7fabbeffd640 1 -- 172.21.15.114:0/932624131 <== mon.2 v1:172.21.15.114:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1594292461 0 0) 0x7fabc0003c60 con 0x7fabcc1422b0 2024-03-29T11:28:35.673 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.679+0000 7fabbeffd640 1 -- 172.21.15.114:0/932624131 <== mon.2 v1:172.21.15.114:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3238987313 0 0) 0x7fabc0084c60 con 0x7fabcc1422b0 2024-03-29T11:28:35.673 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.679+0000 7fabbeffd640 10 client.4532.objecter ms_dispatch 0x7fabcc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:28:35.673 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.679+0000 7fabbeffd640 3 client.4532.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:28:35.673 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.679+0000 7fabbeffd640 3 client.4532.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:28:35.673 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.679+0000 7fabbeffd640 20 client.4532.objecter dump_active .. 0 homeless 2024-03-29T11:28:35.673 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.681+0000 7fabd206f640 10 monclient: get_auth_request con 0x7fabb806dc90 auth_method 0 2024-03-29T11:28:35.673 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.682+0000 7fabbeffd640 10 client.4532.objecter ms_handle_connect 0x7fabb806dc90 2024-03-29T11:28:35.676 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.686+0000 7fabbeffd640 1 -- 172.21.15.114:0/932624131 <== mon.2 v1:172.21.15.114:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fabc0050370 con 0x7fabcc1422b0 2024-03-29T11:28:35.677 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.687+0000 7fabbeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:35.677 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:35.687+0000 7fabbeffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:28:36.007 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:36.017+0000 7fabd42d3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:36.008 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:36.017+0000 7fabd42d3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:36.008 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:36.017+0000 7fabd42d3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:28:36.008 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:36.017+0000 7fabd42d3640 1 -- 172.21.15.114:0/932624131 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fabcc0b2270 con 0x7fabcc1422b0 2024-03-29T11:28:36.008 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:36.018+0000 7fabbeffd640 1 -- 172.21.15.114:0/932624131 <== mon.2 v1:172.21.15.114:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2858224613) 0x7fabc0052470 con 0x7fabcc1422b0 2024-03-29T11:28:36.008 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:36.018+0000 7fabbeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:36.008 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:36.018+0000 7fabbeffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:28:36.010 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T11:28:36.010 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"b0de2b9b-c45a-44c7-b6ee-e83b39a1e108","created":"2024-03-29T11:20:49.420660+0000","modified":"2024-03-29T11:20:51.700167+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:51.700167+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a64785d3-6b11-499a-b706-cc827ab96a74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5882dadb-e7a8-4604-ba9a-418013c106c8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d7e8c409-bc66-4e4a-bbdb-420bad31388e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0682989-3595-4cd6-90a5-ab9111c278e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"67068120-e3ff-4817-bd06-44feb7479a35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09e8b-8a85-4997-bdea-75d754e6ee67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2f0d9ead-f47f-4cb3-92a4-0a5ad23cd831","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fdb6a25f-6bd8-4c65-9b9b-b460ba7673a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:28:36.014 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:36.024+0000 7fabd42d3640 10 librados: watch_flush enter 2024-03-29T11:28:36.014 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:36.024+0000 7fabd42d3640 10 librados: watch_flush exit 2024-03-29T11:28:36.014 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:36.024+0000 7fabd42d3640 20 client.4532.objecter shutdown clearing up homeless session... 2024-03-29T11:28:36.014 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:36.024+0000 7fabd42d3640 10 client.4532.objecter successfully canceled tick 2024-03-29T11:28:36.014 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:36.024+0000 7fabd42d3640 1 -- 172.21.15.114:0/932624131 >> v1:172.21.15.114:6816/4038755055 conn(0x7fabb806dc90 legacy=0x7fabb8070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:36.015 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:36.024+0000 7fabd42d3640 10 monclient: shutdown 2024-03-29T11:28:36.015 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:36.024+0000 7fabd42d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:36.015 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:36.024+0000 7fabd42d3640 1 -- 172.21.15.114:0/932624131 >> v1:172.21.15.114:6790/0 conn(0x7fabcc1422b0 legacy=0x7fabcc1304f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:36.015 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:36.025+0000 7fabd42d3640 1 -- 172.21.15.114:0/932624131 shutdown_connections 2024-03-29T11:28:36.016 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:36.025+0000 7fabd42d3640 1 -- 172.21.15.114:0/932624131 >> 172.21.15.114:0/932624131 conn(0x7fabcc066a00 msgr2=0x7fabcc0b2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:36.016 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:36.026+0000 7fabd42d3640 1 -- 172.21.15.114:0/932624131 shutdown_connections 2024-03-29T11:28:36.016 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:36.026+0000 7fabd42d3640 1 -- 172.21.15.114:0/932624131 wait complete. 2024-03-29T11:28:36.016 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:36.026+0000 7fabd42d3640 1 librados: shutdown 2024-03-29T11:28:36.034 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:28:42.841 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:28:42.994 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.002+0000 7fd45b4c8640 10 monclient: get_monmap_and_config 2024-03-29T11:28:42.994 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.002+0000 7fd45b4c8640 10 monclient: build_initial_monmap 2024-03-29T11:28:42.994 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.002+0000 7fd45b4c8640 10 monclient: monmap: 2024-03-29T11:28:42.994 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:28:42.995 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:28:42.995 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:28:43.003981+0000 2024-03-29T11:28:42.995 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:28:43.003981+0000 2024-03-29T11:28:42.995 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:42.995 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:28:42.995 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:28:42.995 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T11:28:42.995 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T11:28:42.995 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:28:42.995 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.004+0000 7fd45b4c8640 1 Processor -- start 2024-03-29T11:28:42.996 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.005+0000 7fd45b4c8640 1 -- start start 2024-03-29T11:28:42.996 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.005+0000 7fd45b4c8640 10 monclient: init 2024-03-29T11:28:42.996 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.006+0000 7fd45b4c8640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:42.996 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.006+0000 7fd45b4c8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:28:42.997 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.006+0000 7fd45b4c8640 10 monclient(hunting): picked mon.noname-b con 0x7fd4540a9ac0 addr v1:172.21.15.114:6790/0 2024-03-29T11:28:42.997 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.006+0000 7fd45b4c8640 10 monclient(hunting): picked mon.noname-a con 0x7fd45413ee90 addr v1:172.21.15.114:6789/0 2024-03-29T11:28:42.997 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.006+0000 7fd45b4c8640 10 monclient(hunting): picked mon.noname-c con 0x7fd4540af900 addr v1:172.21.15.176:6789/0 2024-03-29T11:28:42.997 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.006+0000 7fd45b4c8640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4540afd10 con 0x7fd45413ee90 2024-03-29T11:28:42.997 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.006+0000 7fd45b4c8640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4540af6f0 con 0x7fd4540af900 2024-03-29T11:28:42.997 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.006+0000 7fd45b4c8640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4540aac80 con 0x7fd4540a9ac0 2024-03-29T11:28:42.997 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.006+0000 7fd45b4c8640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:42.997 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.006+0000 7fd45b4c8640 10 monclient(hunting): authenticate will time out at 1393.188566s 2024-03-29T11:28:43.000 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.010+0000 7fd459264640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7fd4540a9ac0 0x7fd4540adfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:54660/0 (socket says 172.21.15.114:54660) 2024-03-29T11:28:43.000 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.010+0000 7fd459264640 1 -- 172.21.15.114:0/4053647621 learned_addr learned my addr 172.21.15.114:0/4053647621 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:28:43.000 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.010+0000 7fd44bfff640 1 -- 172.21.15.114:0/4053647621 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4102538005 0 0) 0x7fd4540aac80 con 0x7fd4540a9ac0 2024-03-29T11:28:43.001 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.010+0000 7fd44bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:28:43.001 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.010+0000 7fd44bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:43.001 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.010+0000 7fd44bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:43.001 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.010+0000 7fd44bfff640 10 monclient(hunting): my global_id is 4535 2024-03-29T11:28:43.001 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.010+0000 7fd44bfff640 1 -- 172.21.15.114:0/4053647621 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd440003520 con 0x7fd4540a9ac0 2024-03-29T11:28:43.001 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.010+0000 7fd44bfff640 1 -- 172.21.15.114:0/4053647621 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1644340961 0 0) 0x7fd4540af6f0 con 0x7fd4540af900 2024-03-29T11:28:43.001 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.010+0000 7fd44bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:28:43.001 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.010+0000 7fd44bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:43.001 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.010+0000 7fd44bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:43.001 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.010+0000 7fd44bfff640 10 monclient(hunting): my global_id is 4477 2024-03-29T11:28:43.001 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.010+0000 7fd44bfff640 1 -- 172.21.15.114:0/4053647621 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4540aac80 con 0x7fd4540af900 2024-03-29T11:28:43.002 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.011+0000 7fd44bfff640 1 -- 172.21.15.114:0/4053647621 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1353767159 0 0) 0x7fd440003520 con 0x7fd4540a9ac0 2024-03-29T11:28:43.002 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.011+0000 7fd44bfff640 1 -- 172.21.15.114:0/4053647621 >> v1:172.21.15.176:6789/0 conn(0x7fd4540af900 legacy=0x7fd454149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:43.002 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.011+0000 7fd44bfff640 1 -- 172.21.15.114:0/4053647621 >> v1:172.21.15.114:6789/0 conn(0x7fd45413ee90 legacy=0x7fd45413b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:43.002 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.011+0000 7fd44bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:28:43.002 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.011+0000 7fd44bfff640 1 monclient: found mon.noname-b 2024-03-29T11:28:43.002 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.011+0000 7fd44bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:43.002 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.011+0000 7fd44bfff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.114:6790/0 2024-03-29T11:28:43.002 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.011+0000 7fd44bfff640 1 -- 172.21.15.114:0/4053647621 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4540aaf60 con 0x7fd4540a9ac0 2024-03-29T11:28:43.002 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.011+0000 7fd44bfff640 10 monclient: _finish_auth 0 2024-03-29T11:28:43.002 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.011+0000 7fd44bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:28:43.003 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.011+0000 7fd44bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:43.003 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.011+0000 7fd45b4c8640 5 monclient: authenticate success, global_id 4535 2024-03-29T11:28:43.003 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.011+0000 7fd45b4c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:43.003 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.011+0000 7fd44bfff640 1 -- 172.21.15.114:0/4053647621 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fd44c005010 con 0x7fd4540a9ac0 2024-03-29T11:28:43.003 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.011+0000 7fd44bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:43.003 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.011+0000 7fd44bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:28:43.003 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.011+0000 7fd44bfff640 10 monclient: dump: 2024-03-29T11:28:43.003 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:28:43.003 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:28:43.003 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:28:43.003 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:28:43.003 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:43.003 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:28:43.003 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:28:43.004 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:28:43.004 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:28:43.004 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:28:43.004 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.011+0000 7fd45b4c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:43.005 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.011+0000 7fd44bfff640 1 -- 172.21.15.114:0/4053647621 <== mon.2 v1:172.21.15.114:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd44c005230 con 0x7fd4540a9ac0 2024-03-29T11:28:43.005 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.011+0000 7fd44bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:43.005 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.011+0000 7fd45b4c8640 10 monclient: get_monmap_and_config success 2024-03-29T11:28:43.005 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.011+0000 7fd45b4c8640 4 set_mon_vals no callback set 2024-03-29T11:28:43.005 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.012+0000 7fd45b4c8640 10 monclient: shutdown 2024-03-29T11:28:43.005 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.012+0000 7fd45b4c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:43.005 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.012+0000 7fd45b4c8640 1 -- 172.21.15.114:0/4053647621 >> v1:172.21.15.114:6790/0 conn(0x7fd4540a9ac0 legacy=0x7fd4540adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:43.005 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.012+0000 7fd45b4c8640 1 -- 172.21.15.114:0/4053647621 shutdown_connections 2024-03-29T11:28:43.005 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.012+0000 7fd45b4c8640 1 -- 172.21.15.114:0/4053647621 >> 172.21.15.114:0/4053647621 conn(0x7fd4540b35e0 msgr2=0x7fd4540b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:43.005 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.013+0000 7fd45b4c8640 1 -- 172.21.15.114:0/4053647621 shutdown_connections 2024-03-29T11:28:43.006 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.013+0000 7fd45b4c8640 1 -- 172.21.15.114:0/4053647621 wait complete. 2024-03-29T11:28:43.006 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.014+0000 7fd45b4c8640 10 monclient: build_initial_monmap 2024-03-29T11:28:43.006 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.014+0000 7fd45b4c8640 10 monclient: monmap: 2024-03-29T11:28:43.006 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:28:43.006 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:28:43.006 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T11:28:43.006 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T11:28:43.006 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:43.006 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:28:43.006 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:28:43.006 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T11:28:43.006 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T11:28:43.006 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:28:43.006 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.014+0000 7fd45b4c8640 1 librados: starting msgr at 2024-03-29T11:28:43.007 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.014+0000 7fd45b4c8640 1 librados: starting objecter 2024-03-29T11:28:43.007 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.014+0000 7fd45b4c8640 1 Processor -- start 2024-03-29T11:28:43.010 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.015+0000 7fd45b4c8640 1 -- start start 2024-03-29T11:28:43.010 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.015+0000 7fd45b4c8640 1 librados: setting wanted keys 2024-03-29T11:28:43.010 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.015+0000 7fd45b4c8640 1 librados: calling monclient init 2024-03-29T11:28:43.010 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.015+0000 7fd45b4c8640 10 monclient: init 2024-03-29T11:28:43.010 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.015+0000 7fd45b4c8640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:43.010 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.015+0000 7fd45b4c8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:28:43.010 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.015+0000 7fd45b4c8640 10 monclient(hunting): picked mon.noname-c con 0x7fd4540af900 addr v1:172.21.15.114:6790/0 2024-03-29T11:28:43.011 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.015+0000 7fd45b4c8640 10 monclient(hunting): picked mon.noname-a con 0x7fd45413ee90 addr v1:172.21.15.114:6789/0 2024-03-29T11:28:43.011 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.015+0000 7fd45b4c8640 10 monclient(hunting): picked mon.noname-b con 0x7fd45406f930 addr v1:172.21.15.176:6789/0 2024-03-29T11:28:43.011 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.015+0000 7fd45b4c8640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4541c5000 con 0x7fd45413ee90 2024-03-29T11:28:43.011 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.015+0000 7fd45b4c8640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd454070790 con 0x7fd45406f930 2024-03-29T11:28:43.011 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.015+0000 7fd45b4c8640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd454070a70 con 0x7fd4540af900 2024-03-29T11:28:43.011 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.015+0000 7fd45b4c8640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:43.011 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.015+0000 7fd45b4c8640 10 monclient(hunting): authenticate will time out at 1393.197859s 2024-03-29T11:28:43.011 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.018+0000 7fd459264640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7fd4540af900 0x7fd45406ea10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:55376/0 (socket says 172.21.15.114:55376) 2024-03-29T11:28:43.011 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.018+0000 7fd459264640 1 -- 172.21.15.114:0/948323812 learned_addr learned my addr 172.21.15.114:0/948323812 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:28:43.011 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.018+0000 7fd449ffb640 10 client.?.objecter ms_handle_connect 0x7fd4540af900 2024-03-29T11:28:43.011 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.018+0000 7fd449ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:43.011 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.018+0000 7fd449ffb640 10 client.?.objecter ms_handle_connect 0x7fd45413ee90 2024-03-29T11:28:43.011 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.018+0000 7fd449ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:43.012 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.018+0000 7fd449ffb640 1 -- 172.21.15.114:0/948323812 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1321696553 0 0) 0x7fd454070a70 con 0x7fd4540af900 2024-03-29T11:28:43.012 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.018+0000 7fd449ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:28:43.012 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.018+0000 7fd449ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:43.012 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.018+0000 7fd449ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:43.012 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.018+0000 7fd449ffb640 10 monclient(hunting): my global_id is 4538 2024-03-29T11:28:43.012 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.018+0000 7fd449ffb640 1 -- 172.21.15.114:0/948323812 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd438004eb0 con 0x7fd4540af900 2024-03-29T11:28:43.012 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.019+0000 7fd449ffb640 10 client.?.objecter ms_handle_connect 0x7fd45406f930 2024-03-29T11:28:43.012 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.019+0000 7fd449ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:43.012 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.019+0000 7fd449ffb640 1 -- 172.21.15.114:0/948323812 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1368890920 0 0) 0x7fd4541c5000 con 0x7fd45413ee90 2024-03-29T11:28:43.012 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.019+0000 7fd449ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:28:43.012 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.019+0000 7fd449ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:43.012 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.019+0000 7fd449ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:43.012 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.019+0000 7fd449ffb640 10 monclient(hunting): my global_id is 4506 2024-03-29T11:28:43.013 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.019+0000 7fd449ffb640 1 -- 172.21.15.114:0/948323812 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd454070a70 con 0x7fd45413ee90 2024-03-29T11:28:43.013 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.019+0000 7fd449ffb640 1 -- 172.21.15.114:0/948323812 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3542281758 0 0) 0x7fd438004eb0 con 0x7fd4540af900 2024-03-29T11:28:43.013 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.019+0000 7fd449ffb640 1 -- 172.21.15.114:0/948323812 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd4541c5000 con 0x7fd4540af900 2024-03-29T11:28:43.013 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.019+0000 7fd449ffb640 1 -- 172.21.15.114:0/948323812 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fd44c001ad0 con 0x7fd4540af900 2024-03-29T11:28:43.013 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.019+0000 7fd449ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:43.013 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.019+0000 7fd449ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:28:43.013 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.019+0000 7fd449ffb640 10 monclient(hunting): dump: 2024-03-29T11:28:43.013 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:28:43.013 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:28:43.013 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:28:43.013 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:28:43.013 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:43.013 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:28:43.013 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:28:43.014 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:28:43.014 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:28:43.014 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:28:43.018 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.019+0000 7fd449ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:28:43.018 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.019+0000 7fd449ffb640 1 -- 172.21.15.114:0/948323812 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3125353956 0 0) 0x7fd454070790 con 0x7fd45406f930 2024-03-29T11:28:43.018 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.019+0000 7fd449ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:28:43.018 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.019+0000 7fd449ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:43.019 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.019+0000 7fd449ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:43.019 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.019+0000 7fd449ffb640 10 monclient(hunting): my global_id is 4480 2024-03-29T11:28:43.019 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.019+0000 7fd449ffb640 1 -- 172.21.15.114:0/948323812 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd438004eb0 con 0x7fd45406f930 2024-03-29T11:28:43.019 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.019+0000 7fd449ffb640 1 -- 172.21.15.114:0/948323812 <== mon.2 v1:172.21.15.114:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4105424497 0 0) 0x7fd4541c5000 con 0x7fd4540af900 2024-03-29T11:28:43.019 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.019+0000 7fd449ffb640 1 -- 172.21.15.114:0/948323812 >> v1:172.21.15.176:6789/0 conn(0x7fd45406f930 legacy=0x7fd4541e5da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:43.019 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.019+0000 7fd449ffb640 1 -- 172.21.15.114:0/948323812 >> v1:172.21.15.114:6789/0 conn(0x7fd45413ee90 legacy=0x7fd4541c48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:43.019 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.019+0000 7fd449ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:28:43.019 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.019+0000 7fd449ffb640 1 monclient: found mon.c 2024-03-29T11:28:43.019 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.019+0000 7fd449ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:43.019 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.019+0000 7fd449ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:28:43.019 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.019+0000 7fd449ffb640 1 -- 172.21.15.114:0/948323812 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd454070d50 con 0x7fd4540af900 2024-03-29T11:28:43.019 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.019+0000 7fd449ffb640 10 monclient: _finish_auth 0 2024-03-29T11:28:43.019 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.019+0000 7fd449ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:28:43.019 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.019+0000 7fd449ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:43.019 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.019+0000 7fd45b4c8640 5 monclient: authenticate success, global_id 4538 2024-03-29T11:28:43.020 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.019+0000 7fd45b4c8640 10 monclient: _renew_subs 2024-03-29T11:28:43.020 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.020+0000 7fd45b4c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:28:43.020 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.020+0000 7fd45b4c8640 1 -- 172.21.15.114:0/948323812 --> v1:172.21.15.114:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd454072180 con 0x7fd4540af900 2024-03-29T11:28:43.020 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.020+0000 7fd45b4c8640 10 client.4538.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:28:43.020 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.020+0000 7fd45b4c8640 10 monclient: _renew_subs 2024-03-29T11:28:43.020 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.020+0000 7fd45b4c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:28:43.020 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.020+0000 7fd45b4c8640 1 -- 172.21.15.114:0/948323812 --> v1:172.21.15.114:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd454072660 con 0x7fd4540af900 2024-03-29T11:28:43.020 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.020+0000 7fd45b4c8640 1 librados: init done 2024-03-29T11:28:43.020 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.023+0000 7fd449ffb640 1 -- 172.21.15.114:0/948323812 <== mon.2 v1:172.21.15.114:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd44c001fa0 con 0x7fd4540af900 2024-03-29T11:28:43.020 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.023+0000 7fd449ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:43.020 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.023+0000 7fd449ffb640 1 -- 172.21.15.114:0/948323812 <== mon.2 v1:172.21.15.114:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fd44c001790 con 0x7fd4540af900 2024-03-29T11:28:43.020 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.023+0000 7fd449ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:43.020 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.023+0000 7fd449ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:28:43.020 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.023+0000 7fd449ffb640 10 monclient: dump: 2024-03-29T11:28:43.021 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:28:43.021 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:28:43.021 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:28:43.021 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:28:43.021 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:43.021 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:28:43.021 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:28:43.021 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:28:43.021 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:28:43.021 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:28:43.021 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.023+0000 7fd449ffb640 1 -- 172.21.15.114:0/948323812 <== mon.2 v1:172.21.15.114:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1594292461 0 0) 0x7fd44c01b290 con 0x7fd4540af900 2024-03-29T11:28:43.021 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.023+0000 7fd44b7fe640 4 set_mon_vals no callback set 2024-03-29T11:28:43.021 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.023+0000 7fd45b4c8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:43.021 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.023+0000 7fd449ffb640 1 -- 172.21.15.114:0/948323812 <== mon.2 v1:172.21.15.114:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3238987313 0 0) 0x7fd44c084e10 con 0x7fd4540af900 2024-03-29T11:28:43.021 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.023+0000 7fd449ffb640 10 client.4538.objecter ms_dispatch 0x7fd454000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:28:43.022 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.023+0000 7fd449ffb640 3 client.4538.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:28:43.022 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.023+0000 7fd449ffb640 3 client.4538.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:28:43.022 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.023+0000 7fd449ffb640 20 client.4538.objecter dump_active .. 0 homeless 2024-03-29T11:28:43.022 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.024+0000 7fd45b4c8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:43.022 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.024+0000 7fd45b4c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:28:43.022 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.024+0000 7fd45b4c8640 1 -- 172.21.15.114:0/948323812 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd434005080 con 0x7fd4540af900 2024-03-29T11:28:43.022 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.024+0000 7fd458a63640 10 monclient: get_auth_request con 0x7fd43806dbc0 auth_method 0 2024-03-29T11:28:43.022 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.025+0000 7fd449ffb640 10 client.4538.objecter ms_handle_connect 0x7fd43806dbc0 2024-03-29T11:28:43.024 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.034+0000 7fd449ffb640 1 -- 172.21.15.114:0/948323812 <== mon.2 v1:172.21.15.114:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd44c050840 con 0x7fd4540af900 2024-03-29T11:28:43.024 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.034+0000 7fd449ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:43.024 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.034+0000 7fd449ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:28:43.355 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.364+0000 7fd45b4c8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:43.355 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.364+0000 7fd45b4c8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:43.355 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.364+0000 7fd45b4c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:28:43.355 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.364+0000 7fd45b4c8640 1 -- 172.21.15.114:0/948323812 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd434005330 con 0x7fd4540af900 2024-03-29T11:28:43.356 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.366+0000 7fd449ffb640 1 -- 172.21.15.114:0/948323812 <== mon.2 v1:172.21.15.114:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2858224613) 0x7fd44c052770 con 0x7fd4540af900 2024-03-29T11:28:43.356 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.366+0000 7fd449ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:43.356 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.366+0000 7fd449ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:28:43.356 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T11:28:43.357 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"b0de2b9b-c45a-44c7-b6ee-e83b39a1e108","created":"2024-03-29T11:20:49.420660+0000","modified":"2024-03-29T11:20:51.700167+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:51.700167+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a64785d3-6b11-499a-b706-cc827ab96a74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5882dadb-e7a8-4604-ba9a-418013c106c8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d7e8c409-bc66-4e4a-bbdb-420bad31388e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0682989-3595-4cd6-90a5-ab9111c278e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"67068120-e3ff-4817-bd06-44feb7479a35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09e8b-8a85-4997-bdea-75d754e6ee67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2f0d9ead-f47f-4cb3-92a4-0a5ad23cd831","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fdb6a25f-6bd8-4c65-9b9b-b460ba7673a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:28:43.362 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.369+0000 7fd45b4c8640 10 librados: watch_flush enter 2024-03-29T11:28:43.362 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.370+0000 7fd45b4c8640 10 librados: watch_flush exit 2024-03-29T11:28:43.362 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.370+0000 7fd45b4c8640 20 client.4538.objecter shutdown clearing up homeless session... 2024-03-29T11:28:43.362 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.370+0000 7fd45b4c8640 10 client.4538.objecter successfully canceled tick 2024-03-29T11:28:43.362 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.370+0000 7fd45b4c8640 1 -- 172.21.15.114:0/948323812 >> v1:172.21.15.114:6816/4038755055 conn(0x7fd43806dbc0 legacy=0x7fd438070080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:43.362 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.370+0000 7fd45b4c8640 10 monclient: shutdown 2024-03-29T11:28:43.363 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.370+0000 7fd45b4c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:43.363 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.370+0000 7fd45b4c8640 1 -- 172.21.15.114:0/948323812 >> v1:172.21.15.114:6790/0 conn(0x7fd4540af900 legacy=0x7fd45406ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:43.363 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.370+0000 7fd45b4c8640 1 -- 172.21.15.114:0/948323812 shutdown_connections 2024-03-29T11:28:43.363 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.370+0000 7fd45b4c8640 1 -- 172.21.15.114:0/948323812 >> 172.21.15.114:0/948323812 conn(0x7fd4540b35e0 msgr2=0x7fd4540acd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:43.363 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.370+0000 7fd45b4c8640 1 -- 172.21.15.114:0/948323812 shutdown_connections 2024-03-29T11:28:43.364 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.374+0000 7fd45b4c8640 1 -- 172.21.15.114:0/948323812 wait complete. 2024-03-29T11:28:43.364 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:43.374+0000 7fd45b4c8640 1 librados: shutdown 2024-03-29T11:28:43.383 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:28:50.192 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:28:50.337 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.347+0000 7fa8f4d96640 10 monclient: get_monmap_and_config 2024-03-29T11:28:50.338 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.347+0000 7fa8f4d96640 10 monclient: build_initial_monmap 2024-03-29T11:28:50.338 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.347+0000 7fa8f4d96640 10 monclient: monmap: 2024-03-29T11:28:50.338 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:28:50.338 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:28:50.338 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:28:50.348273+0000 2024-03-29T11:28:50.338 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:28:50.348273+0000 2024-03-29T11:28:50.338 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:50.338 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:28:50.338 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:28:50.338 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T11:28:50.338 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T11:28:50.338 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:28:50.339 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.348+0000 7fa8f4d96640 1 Processor -- start 2024-03-29T11:28:50.339 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.349+0000 7fa8f4d96640 1 -- start start 2024-03-29T11:28:50.339 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.349+0000 7fa8f4d96640 10 monclient: init 2024-03-29T11:28:50.340 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.350+0000 7fa8f4d96640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:50.340 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.350+0000 7fa8f4d96640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:28:50.340 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.350+0000 7fa8f4d96640 10 monclient(hunting): picked mon.noname-b con 0x7fa8f01457f0 addr v1:172.21.15.114:6790/0 2024-03-29T11:28:50.340 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.350+0000 7fa8f4d96640 10 monclient(hunting): picked mon.noname-a con 0x7fa8f013f650 addr v1:172.21.15.114:6789/0 2024-03-29T11:28:50.340 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.350+0000 7fa8f4d96640 10 monclient(hunting): picked mon.noname-c con 0x7fa8f00a9d70 addr v1:172.21.15.176:6789/0 2024-03-29T11:28:50.340 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.350+0000 7fa8f4d96640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8f00aaef0 con 0x7fa8f013f650 2024-03-29T11:28:50.340 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.350+0000 7fa8f4d96640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8f0145bd0 con 0x7fa8f00a9d70 2024-03-29T11:28:50.340 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.350+0000 7fa8f4d96640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8f0145eb0 con 0x7fa8f01457f0 2024-03-29T11:28:50.340 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.350+0000 7fa8f4d96640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:50.340 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.350+0000 7fa8f4d96640 10 monclient(hunting): authenticate will time out at 1400.532244s 2024-03-29T11:28:50.341 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.350+0000 7fa8ee59c640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7fa8f01457f0 0x7fa8f013ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:45182/0 (socket says 172.21.15.114:45182) 2024-03-29T11:28:50.341 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.350+0000 7fa8ee59c640 1 -- 172.21.15.114:0/631384501 learned_addr learned my addr 172.21.15.114:0/631384501 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:28:50.341 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.351+0000 7fa8ed59a640 1 -- 172.21.15.114:0/631384501 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3210077337 0 0) 0x7fa8f0145eb0 con 0x7fa8f01457f0 2024-03-29T11:28:50.345 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.353+0000 7fa8ed59a640 10 monclient(hunting): _negotiate 2024-03-29T11:28:50.345 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.353+0000 7fa8ed59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:50.345 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.353+0000 7fa8ed59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:50.345 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.353+0000 7fa8ed59a640 10 monclient(hunting): my global_id is 4541 2024-03-29T11:28:50.345 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.353+0000 7fa8ed59a640 1 -- 172.21.15.114:0/631384501 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8d8003520 con 0x7fa8f01457f0 2024-03-29T11:28:50.346 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.353+0000 7fa8ed59a640 1 -- 172.21.15.114:0/631384501 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3918424828 0 0) 0x7fa8f00aaef0 con 0x7fa8f013f650 2024-03-29T11:28:50.346 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.353+0000 7fa8ed59a640 10 monclient(hunting): _negotiate 2024-03-29T11:28:50.346 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.353+0000 7fa8ed59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:50.346 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.353+0000 7fa8ed59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:50.346 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.353+0000 7fa8ed59a640 10 monclient(hunting): my global_id is 4509 2024-03-29T11:28:50.346 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.353+0000 7fa8ed59a640 1 -- 172.21.15.114:0/631384501 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8f0145eb0 con 0x7fa8f013f650 2024-03-29T11:28:50.346 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.354+0000 7fa8ed59a640 1 -- 172.21.15.114:0/631384501 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 107888020 0 0) 0x7fa8d8003520 con 0x7fa8f01457f0 2024-03-29T11:28:50.346 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.354+0000 7fa8ed59a640 1 -- 172.21.15.114:0/631384501 >> v1:172.21.15.176:6789/0 conn(0x7fa8f00a9d70 legacy=0x7fa8f014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:50.346 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.354+0000 7fa8ed59a640 1 -- 172.21.15.114:0/631384501 >> v1:172.21.15.114:6789/0 conn(0x7fa8f013f650 legacy=0x7fa8f00b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:50.346 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.354+0000 7fa8ed59a640 10 monclient: _finish_hunting 0 2024-03-29T11:28:50.346 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.354+0000 7fa8ed59a640 1 monclient: found mon.noname-b 2024-03-29T11:28:50.346 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.354+0000 7fa8ed59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:50.346 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.354+0000 7fa8ed59a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.114:6790/0 2024-03-29T11:28:50.347 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.354+0000 7fa8ed59a640 1 -- 172.21.15.114:0/631384501 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8f0146340 con 0x7fa8f01457f0 2024-03-29T11:28:50.347 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.354+0000 7fa8ed59a640 10 monclient: _finish_auth 0 2024-03-29T11:28:50.347 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.354+0000 7fa8ed59a640 10 monclient: _check_auth_tickets 2024-03-29T11:28:50.347 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.354+0000 7fa8ed59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:50.347 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.354+0000 7fa8ed59a640 1 -- 172.21.15.114:0/631384501 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fa8e8001990 con 0x7fa8f01457f0 2024-03-29T11:28:50.347 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.354+0000 7fa8ed59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:50.347 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.354+0000 7fa8ed59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:28:50.347 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.354+0000 7fa8ed59a640 10 monclient: dump: 2024-03-29T11:28:50.347 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:28:50.347 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:28:50.347 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:28:50.347 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:28:50.347 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:50.347 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:28:50.347 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:28:50.348 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:28:50.348 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:28:50.348 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:28:50.348 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.354+0000 7fa8f4d96640 5 monclient: authenticate success, global_id 4541 2024-03-29T11:28:50.348 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.354+0000 7fa8f4d96640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:50.348 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.354+0000 7fa8ed59a640 1 -- 172.21.15.114:0/631384501 <== mon.2 v1:172.21.15.114:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa8e8001af0 con 0x7fa8f01457f0 2024-03-29T11:28:50.348 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.354+0000 7fa8ed59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:50.348 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.354+0000 7fa8ed59a640 1 -- 172.21.15.114:0/631384501 <== mon.2 v1:172.21.15.114:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fa8e8001f00 con 0x7fa8f01457f0 2024-03-29T11:28:50.348 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.354+0000 7fa8f4d96640 10 monclient: get_monmap_and_config success 2024-03-29T11:28:50.348 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.354+0000 7fa8f4d96640 4 set_mon_vals no callback set 2024-03-29T11:28:50.348 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.355+0000 7fa8ed59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:50.348 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.355+0000 7fa8ed59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:28:50.348 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.355+0000 7fa8ed59a640 10 monclient: dump: 2024-03-29T11:28:50.348 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:28:50.348 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:28:50.348 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:28:50.349 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:28:50.349 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:50.349 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:28:50.349 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:28:50.349 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:28:50.349 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:28:50.349 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:28:50.351 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.356+0000 7fa8f4d96640 10 monclient: shutdown 2024-03-29T11:28:50.352 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.356+0000 7fa8f4d96640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:50.352 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.356+0000 7fa8f4d96640 1 -- 172.21.15.114:0/631384501 >> v1:172.21.15.114:6790/0 conn(0x7fa8f01457f0 legacy=0x7fa8f013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:50.352 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.356+0000 7fa8f4d96640 1 -- 172.21.15.114:0/631384501 shutdown_connections 2024-03-29T11:28:50.352 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.356+0000 7fa8f4d96640 1 -- 172.21.15.114:0/631384501 >> 172.21.15.114:0/631384501 conn(0x7fa8f0066a00 msgr2=0x7fa8f0066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:50.352 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.357+0000 7fa8f4d96640 1 -- 172.21.15.114:0/631384501 shutdown_connections 2024-03-29T11:28:50.352 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.357+0000 7fa8f4d96640 1 -- 172.21.15.114:0/631384501 wait complete. 2024-03-29T11:28:50.352 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.358+0000 7fa8f4d96640 10 monclient: build_initial_monmap 2024-03-29T11:28:50.352 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.358+0000 7fa8f4d96640 10 monclient: monmap: 2024-03-29T11:28:50.352 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:28:50.352 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:28:50.352 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T11:28:50.352 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T11:28:50.352 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:50.353 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:28:50.353 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:28:50.353 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T11:28:50.353 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T11:28:50.353 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:28:50.353 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.359+0000 7fa8f4d96640 1 librados: starting msgr at 2024-03-29T11:28:50.353 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.359+0000 7fa8f4d96640 1 librados: starting objecter 2024-03-29T11:28:50.353 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.359+0000 7fa8f4d96640 1 Processor -- start 2024-03-29T11:28:50.353 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.359+0000 7fa8f4d96640 1 -- start start 2024-03-29T11:28:50.353 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.359+0000 7fa8f4d96640 1 librados: setting wanted keys 2024-03-29T11:28:50.353 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.359+0000 7fa8f4d96640 1 librados: calling monclient init 2024-03-29T11:28:50.353 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.359+0000 7fa8f4d96640 10 monclient: init 2024-03-29T11:28:50.353 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.360+0000 7fa8f4d96640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:50.354 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.360+0000 7fa8f4d96640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:28:50.354 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.360+0000 7fa8f4d96640 10 monclient(hunting): picked mon.noname-a con 0x7fa8f00a9d70 addr v1:172.21.15.114:6789/0 2024-03-29T11:28:50.356 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.362+0000 7fa8f4d96640 10 monclient(hunting): picked mon.noname-b con 0x7fa8f013f650 addr v1:172.21.15.176:6789/0 2024-03-29T11:28:50.356 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.365+0000 7fa8ee59c640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fa8f00a9d70 0x7fa8f00701c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:36559/0 (socket says 172.21.15.114:36559) 2024-03-29T11:28:50.356 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.365+0000 7fa8eed9d640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7fa8f01457f0 0x7fa8f01e6570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:57576/0 (socket says 172.21.15.114:57576) 2024-03-29T11:28:50.356 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.365+0000 7fa8f4d96640 10 monclient(hunting): picked mon.noname-c con 0x7fa8f01457f0 addr v1:172.21.15.114:6790/0 2024-03-29T11:28:50.356 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.365+0000 7fa8ee59c640 1 -- 172.21.15.114:0/3712098944 learned_addr learned my addr 172.21.15.114:0/3712098944 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:28:50.356 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.365+0000 7fa8f4d96640 1 -- 172.21.15.114:0/3712098944 --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8f00ab1d0 con 0x7fa8f00a9d70 2024-03-29T11:28:50.356 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.365+0000 7fa8f4d96640 1 -- 172.21.15.114:0/3712098944 --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8f0071da0 con 0x7fa8f013f650 2024-03-29T11:28:50.356 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.365+0000 7fa8deffd640 10 client.?.objecter ms_handle_connect 0x7fa8f013f650 2024-03-29T11:28:50.357 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.366+0000 7fa8deffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:50.357 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.366+0000 7fa8f4d96640 1 -- 172.21.15.114:0/3712098944 --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8f00ab870 con 0x7fa8f01457f0 2024-03-29T11:28:50.357 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.366+0000 7fa8f4d96640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:50.357 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.366+0000 7fa8deffd640 1 -- 172.21.15.114:0/3712098944 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2523748538 0 0) 0x7fa8f0071da0 con 0x7fa8f013f650 2024-03-29T11:28:50.357 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.366+0000 7fa8f4d96640 10 monclient(hunting): authenticate will time out at 1400.548382s 2024-03-29T11:28:50.357 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.367+0000 7fa8deffd640 10 monclient(hunting): _negotiate 2024-03-29T11:28:50.357 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.367+0000 7fa8deffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:50.357 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.367+0000 7fa8deffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:50.357 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.367+0000 7fa8deffd640 10 monclient(hunting): my global_id is 4486 2024-03-29T11:28:50.357 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.367+0000 7fa8deffd640 1 -- 172.21.15.114:0/3712098944 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8d0004eb0 con 0x7fa8f013f650 2024-03-29T11:28:50.357 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.367+0000 7fa8deffd640 10 client.?.objecter ms_handle_connect 0x7fa8f01457f0 2024-03-29T11:28:50.358 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.367+0000 7fa8deffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:50.360 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.367+0000 7fa8deffd640 10 client.?.objecter ms_handle_connect 0x7fa8f00a9d70 2024-03-29T11:28:50.360 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.367+0000 7fa8deffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:50.361 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.367+0000 7fa8deffd640 1 -- 172.21.15.114:0/3712098944 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1213250667 0 0) 0x7fa8f00ab870 con 0x7fa8f01457f0 2024-03-29T11:28:50.361 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.367+0000 7fa8deffd640 10 monclient(hunting): _negotiate 2024-03-29T11:28:50.361 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.367+0000 7fa8deffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:50.361 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.367+0000 7fa8deffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:50.361 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.367+0000 7fa8deffd640 10 monclient(hunting): my global_id is 4544 2024-03-29T11:28:50.361 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.367+0000 7fa8deffd640 1 -- 172.21.15.114:0/3712098944 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8f0071da0 con 0x7fa8f01457f0 2024-03-29T11:28:50.361 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.367+0000 7fa8deffd640 1 -- 172.21.15.114:0/3712098944 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1869550869 0 0) 0x7fa8d0004eb0 con 0x7fa8f013f650 2024-03-29T11:28:50.361 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.367+0000 7fa8deffd640 1 -- 172.21.15.114:0/3712098944 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa8f00ab870 con 0x7fa8f013f650 2024-03-29T11:28:50.361 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.367+0000 7fa8deffd640 1 -- 172.21.15.114:0/3712098944 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fa8e4001870 con 0x7fa8f013f650 2024-03-29T11:28:50.361 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.367+0000 7fa8deffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:50.361 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.367+0000 7fa8deffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:28:50.361 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.367+0000 7fa8deffd640 10 monclient(hunting): dump: 2024-03-29T11:28:50.361 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:28:50.361 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:28:50.361 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:28:50.362 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:28:50.362 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:50.362 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:28:50.362 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:28:50.362 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:28:50.362 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:28:50.362 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:28:50.362 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.367+0000 7fa8deffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:28:50.362 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.367+0000 7fa8deffd640 1 -- 172.21.15.114:0/3712098944 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3375304887 0 0) 0x7fa8f0071da0 con 0x7fa8f01457f0 2024-03-29T11:28:50.362 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.367+0000 7fa8deffd640 1 -- 172.21.15.114:0/3712098944 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa8d0004eb0 con 0x7fa8f01457f0 2024-03-29T11:28:50.362 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.367+0000 7fa8deffd640 1 -- 172.21.15.114:0/3712098944 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fa8e0001870 con 0x7fa8f01457f0 2024-03-29T11:28:50.362 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.367+0000 7fa8deffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:50.362 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.367+0000 7fa8deffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:28:50.362 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.367+0000 7fa8deffd640 10 monclient(hunting): dump: 2024-03-29T11:28:50.362 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:28:50.363 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:28:50.363 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:28:50.363 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:28:50.363 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:50.363 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:28:50.363 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:28:50.363 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:28:50.363 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:28:50.363 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:28:50.363 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.368+0000 7fa8deffd640 1 -- 172.21.15.114:0/3712098944 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2995338266 0 0) 0x7fa8f00ab1d0 con 0x7fa8f00a9d70 2024-03-29T11:28:50.363 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.368+0000 7fa8deffd640 10 monclient(hunting): _negotiate 2024-03-29T11:28:50.363 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.368+0000 7fa8deffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:50.363 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.368+0000 7fa8deffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:50.363 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.368+0000 7fa8deffd640 10 monclient(hunting): my global_id is 4512 2024-03-29T11:28:50.363 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.368+0000 7fa8deffd640 1 -- 172.21.15.114:0/3712098944 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8f0071da0 con 0x7fa8f00a9d70 2024-03-29T11:28:50.364 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.368+0000 7fa8deffd640 1 -- 172.21.15.114:0/3712098944 <== mon.1 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 807620052 0 0) 0x7fa8f00ab870 con 0x7fa8f013f650 2024-03-29T11:28:50.364 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.368+0000 7fa8deffd640 1 -- 172.21.15.114:0/3712098944 >> v1:172.21.15.114:6790/0 conn(0x7fa8f01457f0 legacy=0x7fa8f01e6570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:50.364 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.368+0000 7fa8deffd640 1 -- 172.21.15.114:0/3712098944 >> v1:172.21.15.114:6789/0 conn(0x7fa8f00a9d70 legacy=0x7fa8f00701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:50.364 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.368+0000 7fa8deffd640 10 monclient: _finish_hunting 0 2024-03-29T11:28:50.364 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.368+0000 7fa8deffd640 1 monclient: found mon.b 2024-03-29T11:28:50.364 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.368+0000 7fa8deffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:50.364 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.368+0000 7fa8deffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:28:50.364 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.368+0000 7fa8deffd640 1 -- 172.21.15.114:0/3712098944 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8f006fca0 con 0x7fa8f013f650 2024-03-29T11:28:50.364 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.368+0000 7fa8deffd640 10 monclient: _finish_auth 0 2024-03-29T11:28:50.364 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.368+0000 7fa8deffd640 10 monclient: _check_auth_tickets 2024-03-29T11:28:50.364 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.369+0000 7fa8deffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:50.364 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.369+0000 7fa8f4d96640 5 monclient: authenticate success, global_id 4486 2024-03-29T11:28:50.364 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.369+0000 7fa8f4d96640 10 monclient: _renew_subs 2024-03-29T11:28:50.364 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.369+0000 7fa8f4d96640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:28:50.364 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.369+0000 7fa8f4d96640 1 -- 172.21.15.114:0/3712098944 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8f01325d0 con 0x7fa8f013f650 2024-03-29T11:28:50.365 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.369+0000 7fa8f4d96640 10 client.4486.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:28:50.365 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.369+0000 7fa8f4d96640 10 monclient: _renew_subs 2024-03-29T11:28:50.365 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.369+0000 7fa8f4d96640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:28:50.365 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.369+0000 7fa8f4d96640 1 -- 172.21.15.114:0/3712098944 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8f00b0ff0 con 0x7fa8f013f650 2024-03-29T11:28:50.365 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.369+0000 7fa8f4d96640 1 librados: init done 2024-03-29T11:28:50.365 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.369+0000 7fa8deffd640 1 -- 172.21.15.114:0/3712098944 <== mon.1 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa8e40035f0 con 0x7fa8f013f650 2024-03-29T11:28:50.365 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.369+0000 7fa8deffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:50.367 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.369+0000 7fa8ecd99640 4 set_mon_vals no callback set 2024-03-29T11:28:50.367 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.371+0000 7fa8deffd640 1 -- 172.21.15.114:0/3712098944 <== mon.1 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fa8e4001d80 con 0x7fa8f013f650 2024-03-29T11:28:50.367 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.372+0000 7fa8deffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:50.367 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.372+0000 7fa8deffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:28:50.367 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.372+0000 7fa8deffd640 10 monclient: dump: 2024-03-29T11:28:50.367 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:28:50.367 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:28:50.367 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:28:50.367 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:28:50.367 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:50.368 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:28:50.368 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:28:50.368 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:28:50.368 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:28:50.368 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:28:50.368 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.372+0000 7fa8deffd640 1 -- 172.21.15.114:0/3712098944 <== mon.1 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1594292461 0 0) 0x7fa8e4003940 con 0x7fa8f013f650 2024-03-29T11:28:50.368 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.373+0000 7fa8f4d96640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:50.368 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.373+0000 7fa8f4d96640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:50.368 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.373+0000 7fa8f4d96640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:28:50.368 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.373+0000 7fa8f4d96640 1 -- 172.21.15.114:0/3712098944 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8d4005080 con 0x7fa8f013f650 2024-03-29T11:28:50.368 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.373+0000 7fa8deffd640 1 -- 172.21.15.114:0/3712098944 <== mon.1 v1:172.21.15.176:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3238987313 0 0) 0x7fa8e40837b0 con 0x7fa8f013f650 2024-03-29T11:28:50.368 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.373+0000 7fa8ee59c640 10 monclient: get_auth_request con 0x7fa8d006df70 auth_method 0 2024-03-29T11:28:50.368 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.373+0000 7fa8deffd640 10 client.4486.objecter ms_dispatch 0x7fa8f0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:28:50.368 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.373+0000 7fa8deffd640 3 client.4486.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:28:50.368 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.373+0000 7fa8deffd640 3 client.4486.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:28:50.369 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.374+0000 7fa8deffd640 20 client.4486.objecter dump_active .. 0 homeless 2024-03-29T11:28:50.369 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.374+0000 7fa8deffd640 10 client.4486.objecter ms_handle_connect 0x7fa8d006df70 2024-03-29T11:28:50.371 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.381+0000 7fa8deffd640 1 -- 172.21.15.114:0/3712098944 <== mon.1 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa8e404f090 con 0x7fa8f013f650 2024-03-29T11:28:50.371 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.381+0000 7fa8deffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:50.371 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.381+0000 7fa8deffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:28:50.695 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.705+0000 7fa8f4d96640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:50.695 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.705+0000 7fa8f4d96640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:50.696 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.705+0000 7fa8f4d96640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:28:50.696 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.705+0000 7fa8f4d96640 1 -- 172.21.15.114:0/3712098944 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa8d4005600 con 0x7fa8f013f650 2024-03-29T11:28:50.696 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.705+0000 7fa8deffd640 1 -- 172.21.15.114:0/3712098944 <== mon.1 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2858224613) 0x7fa8e4050fc0 con 0x7fa8f013f650 2024-03-29T11:28:50.696 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.705+0000 7fa8deffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:50.696 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.705+0000 7fa8deffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:28:50.697 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T11:28:50.698 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"b0de2b9b-c45a-44c7-b6ee-e83b39a1e108","created":"2024-03-29T11:20:49.420660+0000","modified":"2024-03-29T11:20:51.700167+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:51.700167+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a64785d3-6b11-499a-b706-cc827ab96a74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5882dadb-e7a8-4604-ba9a-418013c106c8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d7e8c409-bc66-4e4a-bbdb-420bad31388e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0682989-3595-4cd6-90a5-ab9111c278e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"67068120-e3ff-4817-bd06-44feb7479a35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09e8b-8a85-4997-bdea-75d754e6ee67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2f0d9ead-f47f-4cb3-92a4-0a5ad23cd831","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fdb6a25f-6bd8-4c65-9b9b-b460ba7673a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:28:50.699 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.709+0000 7fa8f4d96640 10 librados: watch_flush enter 2024-03-29T11:28:50.700 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.710+0000 7fa8f4d96640 10 librados: watch_flush exit 2024-03-29T11:28:50.700 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.710+0000 7fa8f4d96640 20 client.4486.objecter shutdown clearing up homeless session... 2024-03-29T11:28:50.701 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.710+0000 7fa8f4d96640 10 client.4486.objecter successfully canceled tick 2024-03-29T11:28:50.701 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.711+0000 7fa8f4d96640 1 -- 172.21.15.114:0/3712098944 >> v1:172.21.15.114:6816/4038755055 conn(0x7fa8d006df70 legacy=0x7fa8d0070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:50.701 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.711+0000 7fa8f4d96640 10 monclient: shutdown 2024-03-29T11:28:50.701 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.711+0000 7fa8f4d96640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:50.702 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.711+0000 7fa8f4d96640 1 -- 172.21.15.114:0/3712098944 >> v1:172.21.15.176:6789/0 conn(0x7fa8f013f650 legacy=0x7fa8f00708d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:50.702 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.711+0000 7fa8f4d96640 1 -- 172.21.15.114:0/3712098944 shutdown_connections 2024-03-29T11:28:50.702 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.711+0000 7fa8f4d96640 1 -- 172.21.15.114:0/3712098944 >> 172.21.15.114:0/3712098944 conn(0x7fa8f0066a00 msgr2=0x7fa8f013bde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:50.702 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.711+0000 7fa8f4d96640 1 -- 172.21.15.114:0/3712098944 shutdown_connections 2024-03-29T11:28:50.702 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.711+0000 7fa8f4d96640 1 -- 172.21.15.114:0/3712098944 wait complete. 2024-03-29T11:28:50.702 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:50.711+0000 7fa8f4d96640 1 librados: shutdown 2024-03-29T11:28:50.722 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:28:57.528 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:28:57.672 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.681+0000 7faa8e826640 10 monclient: get_monmap_and_config 2024-03-29T11:28:57.672 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.681+0000 7faa8e826640 10 monclient: build_initial_monmap 2024-03-29T11:28:57.672 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.681+0000 7faa8e826640 10 monclient: monmap: 2024-03-29T11:28:57.672 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:28:57.672 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:28:57.672 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:28:57.682392+0000 2024-03-29T11:28:57.672 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:28:57.682392+0000 2024-03-29T11:28:57.672 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:57.672 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:28:57.672 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:28:57.672 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T11:28:57.672 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T11:28:57.672 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:28:57.675 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.684+0000 7faa8e826640 1 Processor -- start 2024-03-29T11:28:57.675 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.685+0000 7faa8e826640 1 -- start start 2024-03-29T11:28:57.675 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.685+0000 7faa8e826640 10 monclient: init 2024-03-29T11:28:57.677 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.686+0000 7faa8e826640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:57.677 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.686+0000 7faa8e826640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:28:57.678 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.686+0000 7faa8e826640 10 monclient(hunting): picked mon.noname-a con 0x7faa80006a70 addr v1:172.21.15.114:6789/0 2024-03-29T11:28:57.678 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.686+0000 7faa8e826640 10 monclient(hunting): picked mon.noname-b con 0x7faa8002b100 addr v1:172.21.15.114:6790/0 2024-03-29T11:28:57.678 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.686+0000 7faa8e826640 10 monclient(hunting): picked mon.noname-c con 0x7faa80029c30 addr v1:172.21.15.176:6789/0 2024-03-29T11:28:57.678 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.686+0000 7faa8e826640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa8002a270 con 0x7faa80006a70 2024-03-29T11:28:57.678 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.686+0000 7faa8e826640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa8002c440 con 0x7faa80029c30 2024-03-29T11:28:57.678 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.686+0000 7faa8e826640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa800c9440 con 0x7faa8002b100 2024-03-29T11:28:57.678 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.686+0000 7faa8e826640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:57.678 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.686+0000 7faa8e826640 10 monclient(hunting): authenticate will time out at 1407.868522s 2024-03-29T11:28:57.678 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.686+0000 7faa8d023640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7faa8002b100 0x7faa800cf100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:52814/0 (socket says 172.21.15.114:52814) 2024-03-29T11:28:57.678 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.686+0000 7faa8d023640 1 -- 172.21.15.114:0/1618398121 learned_addr learned my addr 172.21.15.114:0/1618398121 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:28:57.680 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.689+0000 7faa8c822640 1 -- 172.21.15.114:0/1618398121 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 328717118 0 0) 0x7faa800c9440 con 0x7faa8002b100 2024-03-29T11:28:57.680 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.689+0000 7faa8c822640 10 monclient(hunting): _negotiate 2024-03-29T11:28:57.680 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.689+0000 7faa8c822640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:57.680 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.689+0000 7faa8c822640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:57.680 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.689+0000 7faa8c822640 10 monclient(hunting): my global_id is 4547 2024-03-29T11:28:57.680 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.689+0000 7faa8c822640 1 -- 172.21.15.114:0/1618398121 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa74003520 con 0x7faa8002b100 2024-03-29T11:28:57.680 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.689+0000 7faa8c822640 1 -- 172.21.15.114:0/1618398121 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1392917673 0 0) 0x7faa8002a270 con 0x7faa80006a70 2024-03-29T11:28:57.680 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.689+0000 7faa8c822640 10 monclient(hunting): _negotiate 2024-03-29T11:28:57.683 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.689+0000 7faa8c822640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:57.683 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.689+0000 7faa8c822640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:57.683 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.689+0000 7faa8c822640 10 monclient(hunting): my global_id is 4515 2024-03-29T11:28:57.683 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.689+0000 7faa8c822640 1 -- 172.21.15.114:0/1618398121 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa800c9440 con 0x7faa80006a70 2024-03-29T11:28:57.683 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.689+0000 7faa8c822640 1 -- 172.21.15.114:0/1618398121 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 649857742 0 0) 0x7faa74003520 con 0x7faa8002b100 2024-03-29T11:28:57.683 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.689+0000 7faa8c822640 1 -- 172.21.15.114:0/1618398121 >> v1:172.21.15.176:6789/0 conn(0x7faa80029c30 legacy=0x7faa800d74a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T11:28:57.683 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.689+0000 7faa8c822640 1 -- 172.21.15.114:0/1618398121 >> v1:172.21.15.114:6789/0 conn(0x7faa80006a70 legacy=0x7faa800c8d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:57.684 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.689+0000 7faa8c822640 10 monclient: _finish_hunting 0 2024-03-29T11:28:57.684 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.689+0000 7faa8c822640 1 monclient: found mon.noname-b 2024-03-29T11:28:57.684 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.689+0000 7faa8c822640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:57.684 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.689+0000 7faa8c822640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.114:6790/0 2024-03-29T11:28:57.684 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.689+0000 7faa8c822640 1 -- 172.21.15.114:0/1618398121 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa800c95d0 con 0x7faa8002b100 2024-03-29T11:28:57.684 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.689+0000 7faa8c822640 10 monclient: _finish_auth 0 2024-03-29T11:28:57.684 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.689+0000 7faa8c822640 10 monclient: _check_auth_tickets 2024-03-29T11:28:57.684 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.689+0000 7faa8c822640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:57.684 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.689+0000 7faa8c822640 1 -- 172.21.15.114:0/1618398121 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7faa84001690 con 0x7faa8002b100 2024-03-29T11:28:57.684 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.689+0000 7faa8c822640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:57.684 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.689+0000 7faa8c822640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:28:57.684 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.689+0000 7faa8c822640 10 monclient: dump: 2024-03-29T11:28:57.684 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:28:57.684 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:28:57.685 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:28:57.685 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:28:57.685 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:57.685 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:28:57.685 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:28:57.685 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:28:57.685 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:28:57.685 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:28:57.685 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.690+0000 7faa8c822640 1 -- 172.21.15.114:0/1618398121 <== mon.2 v1:172.21.15.114:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faa84001860 con 0x7faa8002b100 2024-03-29T11:28:57.685 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.690+0000 7faa8c822640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:57.685 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.690+0000 7faa8c822640 1 -- 172.21.15.114:0/1618398121 <== mon.2 v1:172.21.15.114:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7faa84001cb0 con 0x7faa8002b100 2024-03-29T11:28:57.685 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.690+0000 7faa8c822640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:57.685 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.690+0000 7faa8c822640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:28:57.685 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.690+0000 7faa8c822640 10 monclient: dump: 2024-03-29T11:28:57.685 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:28:57.686 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:28:57.686 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:28:57.686 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:28:57.686 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:57.686 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:28:57.686 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:28:57.686 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:28:57.686 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:28:57.686 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:28:57.688 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.693+0000 7faa8e826640 5 monclient: authenticate success, global_id 4547 2024-03-29T11:28:57.688 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.693+0000 7faa8e826640 10 monclient: get_monmap_and_config success 2024-03-29T11:28:57.689 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.693+0000 7faa8e826640 4 set_mon_vals no callback set 2024-03-29T11:28:57.689 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.694+0000 7faa8e826640 10 monclient: shutdown 2024-03-29T11:28:57.689 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.694+0000 7faa8e826640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:57.689 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.694+0000 7faa8e826640 1 -- 172.21.15.114:0/1618398121 >> v1:172.21.15.114:6790/0 conn(0x7faa8002b100 legacy=0x7faa800cf100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:57.689 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.695+0000 7faa8e826640 1 -- 172.21.15.114:0/1618398121 shutdown_connections 2024-03-29T11:28:57.689 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.695+0000 7faa8e826640 1 -- 172.21.15.114:0/1618398121 >> 172.21.15.114:0/1618398121 conn(0x7faa800058f0 msgr2=0x7faa80005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:57.689 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.695+0000 7faa8e826640 1 -- 172.21.15.114:0/1618398121 shutdown_connections 2024-03-29T11:28:57.689 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.695+0000 7faa8e826640 1 -- 172.21.15.114:0/1618398121 wait complete. 2024-03-29T11:28:57.689 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.696+0000 7faa8e826640 10 monclient: build_initial_monmap 2024-03-29T11:28:57.689 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.696+0000 7faa8e826640 10 monclient: monmap: 2024-03-29T11:28:57.689 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:28:57.689 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:28:57.689 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T11:28:57.689 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T11:28:57.689 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:57.690 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:28:57.690 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:28:57.690 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T11:28:57.690 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T11:28:57.690 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:28:57.690 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.697+0000 7faa8e826640 1 librados: starting msgr at 2024-03-29T11:28:57.690 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.697+0000 7faa8e826640 1 librados: starting objecter 2024-03-29T11:28:57.690 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.697+0000 7faa8e826640 1 Processor -- start 2024-03-29T11:28:57.690 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.697+0000 7faa8e826640 1 -- start start 2024-03-29T11:28:57.690 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.697+0000 7faa8e826640 1 librados: setting wanted keys 2024-03-29T11:28:57.690 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.697+0000 7faa8e826640 1 librados: calling monclient init 2024-03-29T11:28:57.690 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.697+0000 7faa8e826640 10 monclient: init 2024-03-29T11:28:57.691 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.698+0000 7faa8e826640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:57.691 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.698+0000 7faa8e826640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:28:57.691 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.698+0000 7faa8e826640 10 monclient(hunting): picked mon.noname-b con 0x7faa80006a70 addr v1:172.21.15.176:6789/0 2024-03-29T11:28:57.691 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.698+0000 7faa8e826640 10 monclient(hunting): picked mon.noname-c con 0x7faa80029c30 addr v1:172.21.15.114:6790/0 2024-03-29T11:28:57.691 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.698+0000 7faa8e826640 10 monclient(hunting): picked mon.noname-a con 0x7faa800c2c90 addr v1:172.21.15.114:6789/0 2024-03-29T11:28:57.692 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.698+0000 7faa8e826640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa800caad0 con 0x7faa800c2c90 2024-03-29T11:28:57.692 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.698+0000 7faa8e826640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa8002d780 con 0x7faa80006a70 2024-03-29T11:28:57.692 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.698+0000 7faa8e826640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa8002f460 con 0x7faa80029c30 2024-03-29T11:28:57.692 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.698+0000 7faa8e826640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:57.692 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.698+0000 7faa8e826640 10 monclient(hunting): authenticate will time out at 1407.880958s 2024-03-29T11:28:57.692 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.698+0000 7faa8d023640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7faa80029c30 0x7faa800c2580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:54254/0 (socket says 172.21.15.114:54254) 2024-03-29T11:28:57.692 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.698+0000 7faa8d023640 1 -- 172.21.15.114:0/2987633815 learned_addr learned my addr 172.21.15.114:0/2987633815 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:28:57.692 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.699+0000 7faa7e7fc640 10 client.?.objecter ms_handle_connect 0x7faa80029c30 2024-03-29T11:28:57.692 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.699+0000 7faa7e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:57.692 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.699+0000 7faa7e7fc640 1 -- 172.21.15.114:0/2987633815 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1816041373 0 0) 0x7faa8002f460 con 0x7faa80029c30 2024-03-29T11:28:57.692 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.699+0000 7faa7e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:28:57.692 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.699+0000 7faa7e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:57.692 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.699+0000 7faa7e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:57.692 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.699+0000 7faa7e7fc640 10 monclient(hunting): my global_id is 4550 2024-03-29T11:28:57.693 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.699+0000 7faa7e7fc640 1 -- 172.21.15.114:0/2987633815 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa6c004eb0 con 0x7faa80029c30 2024-03-29T11:28:57.693 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.700+0000 7faa7e7fc640 1 -- 172.21.15.114:0/2987633815 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1423351572 0 0) 0x7faa6c004eb0 con 0x7faa80029c30 2024-03-29T11:28:57.693 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.700+0000 7faa7e7fc640 1 -- 172.21.15.114:0/2987633815 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faa8002f460 con 0x7faa80029c30 2024-03-29T11:28:57.693 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.700+0000 7faa7e7fc640 1 -- 172.21.15.114:0/2987633815 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7faa84001b70 con 0x7faa80029c30 2024-03-29T11:28:57.693 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.700+0000 7faa7e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:57.693 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.700+0000 7faa7e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:28:57.693 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.700+0000 7faa7e7fc640 10 monclient(hunting): dump: 2024-03-29T11:28:57.693 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:28:57.693 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:28:57.693 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:28:57.693 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:28:57.693 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:57.693 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:28:57.693 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:28:57.693 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:28:57.693 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:28:57.694 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:28:57.694 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.700+0000 7faa7e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:28:57.694 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.703+0000 7faa7e7fc640 1 -- 172.21.15.114:0/2987633815 <== mon.2 v1:172.21.15.114:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4056425477 0 0) 0x7faa8002f460 con 0x7faa80029c30 2024-03-29T11:28:57.694 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.704+0000 7faa7e7fc640 1 -- 172.21.15.114:0/2987633815 >> v1:172.21.15.176:6789/0 conn(0x7faa80006a70 legacy=0x7faa8002d070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:57.694 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.704+0000 7faa7e7fc640 1 -- 172.21.15.114:0/2987633815 >> v1:172.21.15.114:6789/0 conn(0x7faa800c2c90 legacy=0x7faa80160e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:28:57.695 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.704+0000 7faa7e7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:28:57.695 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.704+0000 7faa7e7fc640 1 monclient: found mon.c 2024-03-29T11:28:57.695 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.704+0000 7faa7e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:57.695 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.704+0000 7faa7e7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:28:57.695 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.704+0000 7faa7e7fc640 1 -- 172.21.15.114:0/2987633815 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa8002de10 con 0x7faa80029c30 2024-03-29T11:28:57.695 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.704+0000 7faa7e7fc640 10 monclient: _finish_auth 0 2024-03-29T11:28:57.695 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.704+0000 7faa7e7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:28:57.695 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.704+0000 7faa7e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:57.695 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.704+0000 7faa8e826640 5 monclient: authenticate success, global_id 4550 2024-03-29T11:28:57.695 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.704+0000 7faa8e826640 10 monclient: _renew_subs 2024-03-29T11:28:57.695 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.704+0000 7faa8e826640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:28:57.696 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.704+0000 7faa8e826640 1 -- 172.21.15.114:0/2987633815 --> v1:172.21.15.114:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa8002e0c0 con 0x7faa80029c30 2024-03-29T11:28:57.696 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.704+0000 7faa8e826640 10 client.4550.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:28:57.696 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.704+0000 7faa8e826640 10 monclient: _renew_subs 2024-03-29T11:28:57.696 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.704+0000 7faa8e826640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:28:57.696 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.704+0000 7faa8e826640 1 -- 172.21.15.114:0/2987633815 --> v1:172.21.15.114:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7faa80161740 con 0x7faa80029c30 2024-03-29T11:28:57.696 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.704+0000 7faa8e826640 1 librados: init done 2024-03-29T11:28:57.697 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.705+0000 7faa7e7fc640 1 -- 172.21.15.114:0/2987633815 <== mon.2 v1:172.21.15.114:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faa84004bf0 con 0x7faa80029c30 2024-03-29T11:28:57.697 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.705+0000 7faa7e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:57.697 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.705+0000 7faa7e7fc640 1 -- 172.21.15.114:0/2987633815 <== mon.2 v1:172.21.15.114:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7faa84004f00 con 0x7faa80029c30 2024-03-29T11:28:57.697 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.705+0000 7faa7e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:57.697 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.705+0000 7faa7e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:28:57.697 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.705+0000 7faa7e7fc640 10 monclient: dump: 2024-03-29T11:28:57.698 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:28:57.698 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:28:57.698 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:28:57.698 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:28:57.698 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:57.698 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:28:57.698 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:28:57.698 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:28:57.698 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:28:57.698 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:28:57.698 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.706+0000 7faa8e826640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:57.698 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.706+0000 7faa8e826640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:57.698 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.706+0000 7faa8e826640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:28:57.701 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.706+0000 7faa8e826640 1 -- 172.21.15.114:0/2987633815 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa800c30a0 con 0x7faa80029c30 2024-03-29T11:28:57.701 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.706+0000 7faa7e7fc640 1 -- 172.21.15.114:0/2987633815 <== mon.2 v1:172.21.15.114:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1594292461 0 0) 0x7faa8401aed0 con 0x7faa80029c30 2024-03-29T11:28:57.702 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.707+0000 7faa7f7fe640 4 set_mon_vals no callback set 2024-03-29T11:28:57.702 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.707+0000 7faa7e7fc640 1 -- 172.21.15.114:0/2987633815 <== mon.2 v1:172.21.15.114:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3238987313 0 0) 0x7faa84084bb0 con 0x7faa80029c30 2024-03-29T11:28:57.703 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.707+0000 7faa7e7fc640 10 client.4550.objecter ms_dispatch 0x7faa88000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:28:57.703 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.707+0000 7faa7e7fc640 3 client.4550.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:28:57.703 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.707+0000 7faa7e7fc640 3 client.4550.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:28:57.703 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.707+0000 7faa7e7fc640 20 client.4550.objecter dump_active .. 0 homeless 2024-03-29T11:28:57.703 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.707+0000 7faa8d824640 10 monclient: get_auth_request con 0x7faa6c06db30 auth_method 0 2024-03-29T11:28:57.703 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.708+0000 7faa7e7fc640 10 client.4550.objecter ms_handle_connect 0x7faa6c06db30 2024-03-29T11:28:57.706 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.715+0000 7faa7e7fc640 1 -- 172.21.15.114:0/2987633815 <== mon.2 v1:172.21.15.114:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7faa800c30a0 con 0x7faa80029c30 2024-03-29T11:28:57.706 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.715+0000 7faa7e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:57.706 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:57.715+0000 7faa7e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:28:58.036 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:58.045+0000 7faa8e826640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:58.036 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:58.045+0000 7faa8e826640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:58.036 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:58.045+0000 7faa8e826640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:28:58.036 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:58.045+0000 7faa8e826640 1 -- 172.21.15.114:0/2987633815 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faa800c30a0 con 0x7faa80029c30 2024-03-29T11:28:58.037 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:58.047+0000 7faa7e7fc640 1 -- 172.21.15.114:0/2987633815 <== mon.2 v1:172.21.15.114:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2858224613) 0x7faa800c30a0 con 0x7faa80029c30 2024-03-29T11:28:58.038 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:58.047+0000 7faa7e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:58.038 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:58.047+0000 7faa7e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:28:58.042 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T11:28:58.043 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"b0de2b9b-c45a-44c7-b6ee-e83b39a1e108","created":"2024-03-29T11:20:49.420660+0000","modified":"2024-03-29T11:20:51.700167+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:51.700167+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a64785d3-6b11-499a-b706-cc827ab96a74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5882dadb-e7a8-4604-ba9a-418013c106c8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d7e8c409-bc66-4e4a-bbdb-420bad31388e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0682989-3595-4cd6-90a5-ab9111c278e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"67068120-e3ff-4817-bd06-44feb7479a35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09e8b-8a85-4997-bdea-75d754e6ee67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2f0d9ead-f47f-4cb3-92a4-0a5ad23cd831","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fdb6a25f-6bd8-4c65-9b9b-b460ba7673a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:28:58.043 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:58.051+0000 7faa8e826640 10 librados: watch_flush enter 2024-03-29T11:28:58.043 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:58.052+0000 7faa8e826640 10 librados: watch_flush exit 2024-03-29T11:28:58.044 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:58.052+0000 7faa8e826640 20 client.4550.objecter shutdown clearing up homeless session... 2024-03-29T11:28:58.044 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:58.052+0000 7faa8e826640 10 client.4550.objecter successfully canceled tick 2024-03-29T11:28:58.044 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:58.052+0000 7faa8e826640 1 -- 172.21.15.114:0/2987633815 >> v1:172.21.15.114:6816/4038755055 conn(0x7faa6c06db30 legacy=0x7faa6c06fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:58.044 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:58.052+0000 7faa8e826640 10 monclient: shutdown 2024-03-29T11:28:58.044 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:58.052+0000 7faa8e826640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:58.044 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:58.052+0000 7faa8e826640 1 -- 172.21.15.114:0/2987633815 >> v1:172.21.15.114:6790/0 conn(0x7faa80029c30 legacy=0x7faa800c2580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:58.044 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:58.052+0000 7faa8e826640 1 -- 172.21.15.114:0/2987633815 shutdown_connections 2024-03-29T11:28:58.044 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:58.052+0000 7faa8e826640 1 -- 172.21.15.114:0/2987633815 >> 172.21.15.114:0/2987633815 conn(0x7faa800058f0 msgr2=0x7faa800374a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:58.044 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:58.052+0000 7faa8e826640 1 -- 172.21.15.114:0/2987633815 shutdown_connections 2024-03-29T11:28:58.044 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:58.052+0000 7faa8e826640 1 -- 172.21.15.114:0/2987633815 wait complete. 2024-03-29T11:28:58.044 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:28:58.052+0000 7faa8e826640 1 librados: shutdown 2024-03-29T11:28:58.060 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:29:04.867 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:29:05.014 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.023+0000 7fbc1c20f640 10 monclient: get_monmap_and_config 2024-03-29T11:29:05.015 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.023+0000 7fbc1c20f640 10 monclient: build_initial_monmap 2024-03-29T11:29:05.015 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.023+0000 7fbc1c20f640 10 monclient: monmap: 2024-03-29T11:29:05.015 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:29:05.015 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:29:05.015 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:29:05.024946+0000 2024-03-29T11:29:05.015 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:29:05.024946+0000 2024-03-29T11:29:05.015 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:05.015 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:29:05.015 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:29:05.015 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T11:29:05.015 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T11:29:05.015 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:29:05.016 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.024+0000 7fbc1c20f640 1 Processor -- start 2024-03-29T11:29:05.019 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.025+0000 7fbc1c20f640 1 -- start start 2024-03-29T11:29:05.022 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.025+0000 7fbc1c20f640 10 monclient: init 2024-03-29T11:29:05.022 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.025+0000 7fbc1c20f640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:05.022 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.025+0000 7fbc1c20f640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:29:05.022 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.025+0000 7fbc1c20f640 10 monclient(hunting): picked mon.noname-a con 0x7fbc141457f0 addr v1:172.21.15.114:6789/0 2024-03-29T11:29:05.022 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.026+0000 7fbc1c20f640 10 monclient(hunting): picked mon.noname-c con 0x7fbc1413f650 addr v1:172.21.15.176:6789/0 2024-03-29T11:29:05.022 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.026+0000 7fbc1c20f640 10 monclient(hunting): picked mon.noname-b con 0x7fbc140a9d70 addr v1:172.21.15.114:6790/0 2024-03-29T11:29:05.022 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.026+0000 7fbc1c20f640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc140aaef0 con 0x7fbc141457f0 2024-03-29T11:29:05.022 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.026+0000 7fbc1c20f640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc14145bd0 con 0x7fbc1413f650 2024-03-29T11:29:05.022 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.026+0000 7fbc1c20f640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc14145eb0 con 0x7fbc140a9d70 2024-03-29T11:29:05.022 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.026+0000 7fbc1c20f640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:05.023 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.026+0000 7fbc1c20f640 10 monclient(hunting): authenticate will time out at 1415.208143s 2024-03-29T11:29:05.023 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.028+0000 7fbc1a7ac640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7fbc140a9d70 0x7fbc1414d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:39356/0 (socket says 172.21.15.114:39356) 2024-03-29T11:29:05.023 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.028+0000 7fbc1a7ac640 1 -- 172.21.15.114:0/2228388746 learned_addr learned my addr 172.21.15.114:0/2228388746 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:29:05.023 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.029+0000 7fbc18fa9640 1 -- 172.21.15.114:0/2228388746 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3313466128 0 0) 0x7fbc14145eb0 con 0x7fbc140a9d70 2024-03-29T11:29:05.023 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.029+0000 7fbc18fa9640 10 monclient(hunting): _negotiate 2024-03-29T11:29:05.023 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.029+0000 7fbc18fa9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:05.023 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.029+0000 7fbc18fa9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:05.023 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.029+0000 7fbc18fa9640 10 monclient(hunting): my global_id is 4553 2024-03-29T11:29:05.023 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.029+0000 7fbc18fa9640 1 -- 172.21.15.114:0/2228388746 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc00003520 con 0x7fbc140a9d70 2024-03-29T11:29:05.024 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.032+0000 7fbc18fa9640 1 -- 172.21.15.114:0/2228388746 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1898496023 0 0) 0x7fbc00003520 con 0x7fbc140a9d70 2024-03-29T11:29:05.024 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.032+0000 7fbc18fa9640 1 -- 172.21.15.114:0/2228388746 >> v1:172.21.15.176:6789/0 conn(0x7fbc1413f650 legacy=0x7fbc140b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:29:05.024 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.032+0000 7fbc18fa9640 1 -- 172.21.15.114:0/2228388746 >> v1:172.21.15.114:6789/0 conn(0x7fbc141457f0 legacy=0x7fbc1413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:05.024 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.032+0000 7fbc18fa9640 10 monclient: _finish_hunting 0 2024-03-29T11:29:05.024 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.032+0000 7fbc18fa9640 1 monclient: found mon.noname-b 2024-03-29T11:29:05.024 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.032+0000 7fbc18fa9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:05.024 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.032+0000 7fbc18fa9640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.114:6790/0 2024-03-29T11:29:05.024 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.032+0000 7fbc18fa9640 1 -- 172.21.15.114:0/2228388746 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc14146340 con 0x7fbc140a9d70 2024-03-29T11:29:05.024 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.032+0000 7fbc18fa9640 10 monclient: _finish_auth 0 2024-03-29T11:29:05.024 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.032+0000 7fbc18fa9640 10 monclient: _check_auth_tickets 2024-03-29T11:29:05.024 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.032+0000 7fbc18fa9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:05.025 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.032+0000 7fbc18fa9640 1 -- 172.21.15.114:0/2228388746 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fbc04001690 con 0x7fbc140a9d70 2024-03-29T11:29:05.025 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.032+0000 7fbc1c20f640 5 monclient: authenticate success, global_id 4553 2024-03-29T11:29:05.025 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.032+0000 7fbc1c20f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:05.025 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.032+0000 7fbc18fa9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:05.025 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.032+0000 7fbc18fa9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:29:05.025 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.032+0000 7fbc18fa9640 10 monclient: dump: 2024-03-29T11:29:05.025 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:29:05.025 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:29:05.025 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:29:05.025 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:29:05.025 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:05.025 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:29:05.025 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:29:05.025 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:29:05.025 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:29:05.026 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:29:05.026 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.032+0000 7fbc1c20f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:05.026 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.032+0000 7fbc18fa9640 1 -- 172.21.15.114:0/2228388746 <== mon.2 v1:172.21.15.114:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbc04001860 con 0x7fbc140a9d70 2024-03-29T11:29:05.026 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.032+0000 7fbc18fa9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:05.026 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.033+0000 7fbc1c20f640 10 monclient: get_monmap_and_config success 2024-03-29T11:29:05.026 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.033+0000 7fbc1c20f640 4 set_mon_vals no callback set 2024-03-29T11:29:05.026 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.033+0000 7fbc18fa9640 1 -- 172.21.15.114:0/2228388746 <== mon.2 v1:172.21.15.114:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fbc04001cb0 con 0x7fbc140a9d70 2024-03-29T11:29:05.026 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.033+0000 7fbc18fa9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:05.026 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.033+0000 7fbc18fa9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:29:05.026 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.033+0000 7fbc18fa9640 10 monclient: dump: 2024-03-29T11:29:05.026 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:29:05.026 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:29:05.026 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:29:05.026 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:29:05.027 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:05.027 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:29:05.027 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:29:05.027 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:29:05.027 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:29:05.027 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:29:05.027 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.033+0000 7fbc1c20f640 10 monclient: shutdown 2024-03-29T11:29:05.028 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.033+0000 7fbc1c20f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:05.028 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.033+0000 7fbc1c20f640 1 -- 172.21.15.114:0/2228388746 >> v1:172.21.15.114:6790/0 conn(0x7fbc140a9d70 legacy=0x7fbc1414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:05.028 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.034+0000 7fbc1c20f640 1 -- 172.21.15.114:0/2228388746 shutdown_connections 2024-03-29T11:29:05.028 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.034+0000 7fbc1c20f640 1 -- 172.21.15.114:0/2228388746 >> 172.21.15.114:0/2228388746 conn(0x7fbc14066a00 msgr2=0x7fbc14066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:05.028 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.034+0000 7fbc1c20f640 1 -- 172.21.15.114:0/2228388746 shutdown_connections 2024-03-29T11:29:05.029 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.034+0000 7fbc1c20f640 1 -- 172.21.15.114:0/2228388746 wait complete. 2024-03-29T11:29:05.029 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.035+0000 7fbc1c20f640 10 monclient: build_initial_monmap 2024-03-29T11:29:05.036 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.035+0000 7fbc1c20f640 10 monclient: monmap: 2024-03-29T11:29:05.036 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:29:05.036 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:29:05.036 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T11:29:05.036 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T11:29:05.036 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:05.036 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:29:05.036 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:29:05.036 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T11:29:05.036 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T11:29:05.036 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:29:05.036 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.036+0000 7fbc1c20f640 1 librados: starting msgr at 2024-03-29T11:29:05.036 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.036+0000 7fbc1c20f640 1 librados: starting objecter 2024-03-29T11:29:05.037 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.036+0000 7fbc1c20f640 1 Processor -- start 2024-03-29T11:29:05.037 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.036+0000 7fbc1c20f640 1 -- start start 2024-03-29T11:29:05.037 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.036+0000 7fbc1c20f640 1 librados: setting wanted keys 2024-03-29T11:29:05.037 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.036+0000 7fbc1c20f640 1 librados: calling monclient init 2024-03-29T11:29:05.037 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.036+0000 7fbc1c20f640 10 monclient: init 2024-03-29T11:29:05.037 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.037+0000 7fbc1c20f640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:05.037 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.037+0000 7fbc1c20f640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:29:05.037 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.037+0000 7fbc1c20f640 10 monclient(hunting): picked mon.noname-c con 0x7fbc140a9d70 addr v1:172.21.15.114:6790/0 2024-03-29T11:29:05.037 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.037+0000 7fbc1c20f640 10 monclient(hunting): picked mon.noname-a con 0x7fbc1413f650 addr v1:172.21.15.114:6789/0 2024-03-29T11:29:05.037 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.037+0000 7fbc1c20f640 10 monclient(hunting): picked mon.noname-b con 0x7fbc141457f0 addr v1:172.21.15.176:6789/0 2024-03-29T11:29:05.037 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.037+0000 7fbc1c20f640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc140ac250 con 0x7fbc1413f650 2024-03-29T11:29:05.037 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.037+0000 7fbc1c20f640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc140ab8e0 con 0x7fbc141457f0 2024-03-29T11:29:05.037 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.037+0000 7fbc1c20f640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc14137740 con 0x7fbc140a9d70 2024-03-29T11:29:05.037 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.037+0000 7fbc1c20f640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:05.038 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.037+0000 7fbc1c20f640 10 monclient(hunting): authenticate will time out at 1415.219619s 2024-03-29T11:29:05.038 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.038+0000 7fbc19fab640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7fbc140a9d70 0x7fbc14144ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:41262/0 (socket says 172.21.15.114:41262) 2024-03-29T11:29:05.038 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.040+0000 7fbc19fab640 1 -- 172.21.15.114:0/1806168699 learned_addr learned my addr 172.21.15.114:0/1806168699 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:29:05.038 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.041+0000 7fbc0a7fc640 10 client.?.objecter ms_handle_connect 0x7fbc140a9d70 2024-03-29T11:29:05.038 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.041+0000 7fbc0a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:05.038 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.041+0000 7fbc0a7fc640 1 -- 172.21.15.114:0/1806168699 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3952550539 0 0) 0x7fbc14137740 con 0x7fbc140a9d70 2024-03-29T11:29:05.038 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.041+0000 7fbc0a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:29:05.038 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.041+0000 7fbc0a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:05.039 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.041+0000 7fbc0a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:05.039 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.041+0000 7fbc0a7fc640 10 monclient(hunting): my global_id is 4556 2024-03-29T11:29:05.039 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.041+0000 7fbc0a7fc640 1 -- 172.21.15.114:0/1806168699 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbbf8004eb0 con 0x7fbc140a9d70 2024-03-29T11:29:05.039 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.042+0000 7fbc0a7fc640 1 -- 172.21.15.114:0/1806168699 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3493060452 0 0) 0x7fbbf8004eb0 con 0x7fbc140a9d70 2024-03-29T11:29:05.039 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.042+0000 7fbc0a7fc640 1 -- 172.21.15.114:0/1806168699 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbc14137740 con 0x7fbc140a9d70 2024-03-29T11:29:05.039 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.042+0000 7fbc0a7fc640 1 -- 172.21.15.114:0/1806168699 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fbc0c001d10 con 0x7fbc140a9d70 2024-03-29T11:29:05.039 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.042+0000 7fbc0a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:05.039 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.042+0000 7fbc0a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:29:05.039 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.042+0000 7fbc0a7fc640 10 monclient(hunting): dump: 2024-03-29T11:29:05.039 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:29:05.039 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:29:05.039 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:29:05.039 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:29:05.040 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:05.040 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:29:05.040 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:29:05.040 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:29:05.040 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:29:05.040 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:29:05.041 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.042+0000 7fbc0a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:29:05.041 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.042+0000 7fbc0a7fc640 10 client.?.objecter ms_handle_connect 0x7fbc1413f650 2024-03-29T11:29:05.041 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.042+0000 7fbc0a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:05.041 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.042+0000 7fbc0a7fc640 1 -- 172.21.15.114:0/1806168699 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3486989279 0 0) 0x7fbc140ac250 con 0x7fbc1413f650 2024-03-29T11:29:05.041 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.042+0000 7fbc0a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:29:05.041 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.042+0000 7fbc0a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:05.041 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.042+0000 7fbc0a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:05.041 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.042+0000 7fbc0a7fc640 10 monclient(hunting): my global_id is 4521 2024-03-29T11:29:05.041 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.042+0000 7fbc0a7fc640 1 -- 172.21.15.114:0/1806168699 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbbf8004eb0 con 0x7fbc1413f650 2024-03-29T11:29:05.041 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.042+0000 7fbc0a7fc640 1 -- 172.21.15.114:0/1806168699 <== mon.2 v1:172.21.15.114:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4141656576 0 0) 0x7fbc14137740 con 0x7fbc140a9d70 2024-03-29T11:29:05.041 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.042+0000 7fbc0a7fc640 1 -- 172.21.15.114:0/1806168699 >> v1:172.21.15.176:6789/0 conn(0x7fbc141457f0 legacy=0x7fbc141f3380 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:29:05.041 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.043+0000 7fbc0a7fc640 1 -- 172.21.15.114:0/1806168699 >> v1:172.21.15.114:6789/0 conn(0x7fbc1413f650 legacy=0x7fbc14142670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:05.042 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.043+0000 7fbc0a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:29:05.042 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.043+0000 7fbc0a7fc640 1 monclient: found mon.c 2024-03-29T11:29:05.042 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.043+0000 7fbc0a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:05.042 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.043+0000 7fbc0a7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:29:05.042 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.043+0000 7fbc0a7fc640 1 -- 172.21.15.114:0/1806168699 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc140ac4a0 con 0x7fbc140a9d70 2024-03-29T11:29:05.042 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.043+0000 7fbc0a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:29:05.042 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.043+0000 7fbc0a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:29:05.042 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.043+0000 7fbc0a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:05.042 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.043+0000 7fbc1c20f640 5 monclient: authenticate success, global_id 4556 2024-03-29T11:29:05.042 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.043+0000 7fbc1c20f640 10 monclient: _renew_subs 2024-03-29T11:29:05.042 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.043+0000 7fbc1c20f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:29:05.042 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.043+0000 7fbc1c20f640 1 -- 172.21.15.114:0/1806168699 --> v1:172.21.15.114:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc140ac780 con 0x7fbc140a9d70 2024-03-29T11:29:05.042 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.043+0000 7fbc1c20f640 10 client.4556.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:29:05.042 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.043+0000 7fbc1c20f640 10 monclient: _renew_subs 2024-03-29T11:29:05.043 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.043+0000 7fbc1c20f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:29:05.043 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.043+0000 7fbc1c20f640 1 -- 172.21.15.114:0/1806168699 --> v1:172.21.15.114:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc14142d80 con 0x7fbc140a9d70 2024-03-29T11:29:05.043 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.044+0000 7fbc1c20f640 1 librados: init done 2024-03-29T11:29:05.043 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.044+0000 7fbc0a7fc640 1 -- 172.21.15.114:0/1806168699 <== mon.2 v1:172.21.15.114:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbc0c001e70 con 0x7fbc140a9d70 2024-03-29T11:29:05.043 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.044+0000 7fbc0a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:05.043 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.044+0000 7fbc0a7fc640 1 -- 172.21.15.114:0/1806168699 <== mon.2 v1:172.21.15.114:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fbc0c003c40 con 0x7fbc140a9d70 2024-03-29T11:29:05.043 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.044+0000 7fbc0a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:05.043 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.044+0000 7fbc0a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:29:05.043 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.044+0000 7fbc0a7fc640 10 monclient: dump: 2024-03-29T11:29:05.043 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:29:05.043 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:29:05.043 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:29:05.043 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:29:05.043 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:05.044 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:29:05.044 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:29:05.044 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:29:05.044 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:29:05.044 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:29:05.045 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.045+0000 7fbc0bfff640 4 set_mon_vals no callback set 2024-03-29T11:29:05.045 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.047+0000 7fbc1c20f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:05.045 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.047+0000 7fbc1c20f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:05.045 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.047+0000 7fbc1c20f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:29:05.045 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.047+0000 7fbc1c20f640 1 -- 172.21.15.114:0/1806168699 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbfc005080 con 0x7fbc140a9d70 2024-03-29T11:29:05.045 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.048+0000 7fbc0a7fc640 1 -- 172.21.15.114:0/1806168699 <== mon.2 v1:172.21.15.114:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1594292461 0 0) 0x7fbc0c003e80 con 0x7fbc140a9d70 2024-03-29T11:29:05.045 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.048+0000 7fbc0a7fc640 1 -- 172.21.15.114:0/1806168699 <== mon.2 v1:172.21.15.114:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3238987313 0 0) 0x7fbc0c084e00 con 0x7fbc140a9d70 2024-03-29T11:29:05.045 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.048+0000 7fbc0a7fc640 10 client.4556.objecter ms_dispatch 0x7fbc14000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:29:05.045 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.048+0000 7fbc0a7fc640 3 client.4556.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:29:05.045 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.048+0000 7fbc0a7fc640 3 client.4556.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:29:05.045 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.048+0000 7fbc0a7fc640 20 client.4556.objecter dump_active .. 0 homeless 2024-03-29T11:29:05.046 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.050+0000 7fbc197aa640 10 monclient: get_auth_request con 0x7fbbf806dad0 auth_method 0 2024-03-29T11:29:05.046 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.050+0000 7fbc0a7fc640 10 client.4556.objecter ms_handle_connect 0x7fbbf806dad0 2024-03-29T11:29:05.048 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.057+0000 7fbc0a7fc640 1 -- 172.21.15.114:0/1806168699 <== mon.2 v1:172.21.15.114:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbc0c0506e0 con 0x7fbc140a9d70 2024-03-29T11:29:05.048 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.057+0000 7fbc0a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:05.048 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.057+0000 7fbc0a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:29:05.377 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.385+0000 7fbc1c20f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:05.377 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.385+0000 7fbc1c20f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:05.377 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.385+0000 7fbc1c20f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:29:05.377 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.385+0000 7fbc1c20f640 1 -- 172.21.15.114:0/1806168699 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbbfc005330 con 0x7fbc140a9d70 2024-03-29T11:29:05.379 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.389+0000 7fbc0a7fc640 1 -- 172.21.15.114:0/1806168699 <== mon.2 v1:172.21.15.114:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2858224613) 0x7fbc0c052610 con 0x7fbc140a9d70 2024-03-29T11:29:05.379 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.389+0000 7fbc0a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:05.379 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.389+0000 7fbc0a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:29:05.381 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T11:29:05.381 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"b0de2b9b-c45a-44c7-b6ee-e83b39a1e108","created":"2024-03-29T11:20:49.420660+0000","modified":"2024-03-29T11:20:51.700167+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:51.700167+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a64785d3-6b11-499a-b706-cc827ab96a74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5882dadb-e7a8-4604-ba9a-418013c106c8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d7e8c409-bc66-4e4a-bbdb-420bad31388e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0682989-3595-4cd6-90a5-ab9111c278e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"67068120-e3ff-4817-bd06-44feb7479a35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09e8b-8a85-4997-bdea-75d754e6ee67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2f0d9ead-f47f-4cb3-92a4-0a5ad23cd831","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fdb6a25f-6bd8-4c65-9b9b-b460ba7673a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:29:05.382 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.392+0000 7fbc1c20f640 10 librados: watch_flush enter 2024-03-29T11:29:05.383 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.392+0000 7fbc1c20f640 10 librados: watch_flush exit 2024-03-29T11:29:05.383 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.392+0000 7fbc1c20f640 20 client.4556.objecter shutdown clearing up homeless session... 2024-03-29T11:29:05.383 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.392+0000 7fbc1c20f640 10 client.4556.objecter successfully canceled tick 2024-03-29T11:29:05.383 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.392+0000 7fbc1c20f640 1 -- 172.21.15.114:0/1806168699 >> v1:172.21.15.114:6816/4038755055 conn(0x7fbbf806dad0 legacy=0x7fbbf806ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:05.383 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.392+0000 7fbc1c20f640 10 monclient: shutdown 2024-03-29T11:29:05.383 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.392+0000 7fbc1c20f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:05.383 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.392+0000 7fbc1c20f640 1 -- 172.21.15.114:0/1806168699 >> v1:172.21.15.114:6790/0 conn(0x7fbc140a9d70 legacy=0x7fbc14144ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:05.384 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.392+0000 7fbc1c20f640 1 -- 172.21.15.114:0/1806168699 shutdown_connections 2024-03-29T11:29:05.384 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.392+0000 7fbc1c20f640 1 -- 172.21.15.114:0/1806168699 >> 172.21.15.114:0/1806168699 conn(0x7fbc14066a00 msgr2=0x7fbc1413aa40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:05.384 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.393+0000 7fbc1c20f640 1 -- 172.21.15.114:0/1806168699 shutdown_connections 2024-03-29T11:29:05.384 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.393+0000 7fbc1c20f640 1 -- 172.21.15.114:0/1806168699 wait complete. 2024-03-29T11:29:05.384 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:05.393+0000 7fbc1c20f640 1 librados: shutdown 2024-03-29T11:29:05.400 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:29:12.207 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:29:12.362 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.371+0000 7f5b30539640 10 monclient: get_monmap_and_config 2024-03-29T11:29:12.362 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.371+0000 7f5b30539640 10 monclient: build_initial_monmap 2024-03-29T11:29:12.362 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.371+0000 7f5b30539640 10 monclient: monmap: 2024-03-29T11:29:12.362 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:29:12.362 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:29:12.362 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:29:12.372178+0000 2024-03-29T11:29:12.362 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:29:12.372178+0000 2024-03-29T11:29:12.362 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:12.362 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:29:12.363 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:29:12.363 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T11:29:12.363 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T11:29:12.363 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:29:12.363 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.372+0000 7f5b30539640 1 Processor -- start 2024-03-29T11:29:12.363 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.372+0000 7f5b30539640 1 -- start start 2024-03-29T11:29:12.363 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.372+0000 7f5b30539640 10 monclient: init 2024-03-29T11:29:12.366 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.373+0000 7f5b30539640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:12.366 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.373+0000 7f5b30539640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:29:12.366 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.373+0000 7f5b30539640 10 monclient(hunting): picked mon.noname-a con 0x7f5b281457f0 addr v1:172.21.15.114:6789/0 2024-03-29T11:29:12.367 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.373+0000 7f5b30539640 10 monclient(hunting): picked mon.noname-b con 0x7f5b2813f650 addr v1:172.21.15.114:6790/0 2024-03-29T11:29:12.367 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.373+0000 7f5b30539640 10 monclient(hunting): picked mon.noname-c con 0x7f5b280a9d70 addr v1:172.21.15.176:6789/0 2024-03-29T11:29:12.367 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.373+0000 7f5b30539640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b280aaef0 con 0x7f5b281457f0 2024-03-29T11:29:12.367 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.373+0000 7f5b30539640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b28145bd0 con 0x7f5b280a9d70 2024-03-29T11:29:12.367 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.373+0000 7f5b30539640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b28145eb0 con 0x7f5b2813f650 2024-03-29T11:29:12.367 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.373+0000 7f5b30539640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:12.367 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.373+0000 7f5b30539640 10 monclient(hunting): authenticate will time out at 1422.555499s 2024-03-29T11:29:12.370 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.379+0000 7f5b2dad4640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f5b2813f650 0x7f5b280b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:34629/0 (socket says 172.21.15.114:34629) 2024-03-29T11:29:12.370 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.379+0000 7f5b2e2d5640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f5b281457f0 0x7f5b2813ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:44333/0 (socket says 172.21.15.114:44333) 2024-03-29T11:29:12.370 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.379+0000 7f5b2e2d5640 1 -- 172.21.15.114:0/3650761085 learned_addr learned my addr 172.21.15.114:0/3650761085 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:29:12.371 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.379+0000 7f5b2d2d3640 1 -- 172.21.15.114:0/3650761085 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4158089110 0 0) 0x7f5b28145eb0 con 0x7f5b2813f650 2024-03-29T11:29:12.371 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.379+0000 7f5b2d2d3640 10 monclient(hunting): _negotiate 2024-03-29T11:29:12.371 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.379+0000 7f5b2d2d3640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:12.371 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.379+0000 7f5b2d2d3640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:12.371 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.379+0000 7f5b2d2d3640 10 monclient(hunting): my global_id is 4559 2024-03-29T11:29:12.371 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.379+0000 7f5b2d2d3640 1 -- 172.21.15.114:0/3650761085 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b14003520 con 0x7f5b2813f650 2024-03-29T11:29:12.378 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.383+0000 7f5b2d2d3640 1 -- 172.21.15.114:0/3650761085 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1939762885 0 0) 0x7f5b280aaef0 con 0x7f5b281457f0 2024-03-29T11:29:12.378 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.383+0000 7f5b2d2d3640 10 monclient(hunting): _negotiate 2024-03-29T11:29:12.378 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.383+0000 7f5b2d2d3640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:12.378 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.383+0000 7f5b2d2d3640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:12.378 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.383+0000 7f5b2d2d3640 10 monclient(hunting): my global_id is 4524 2024-03-29T11:29:12.378 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.383+0000 7f5b2d2d3640 1 -- 172.21.15.114:0/3650761085 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b28145eb0 con 0x7f5b281457f0 2024-03-29T11:29:12.378 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.383+0000 7f5b2d2d3640 1 -- 172.21.15.114:0/3650761085 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4192513655 0 0) 0x7f5b14003520 con 0x7f5b2813f650 2024-03-29T11:29:12.378 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.383+0000 7f5b2d2d3640 1 -- 172.21.15.114:0/3650761085 >> v1:172.21.15.176:6789/0 conn(0x7f5b280a9d70 legacy=0x7f5b2814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:29:12.378 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.383+0000 7f5b2d2d3640 1 -- 172.21.15.114:0/3650761085 >> v1:172.21.15.114:6789/0 conn(0x7f5b281457f0 legacy=0x7f5b2813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:12.378 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.383+0000 7f5b2d2d3640 10 monclient: _finish_hunting 0 2024-03-29T11:29:12.378 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.383+0000 7f5b2d2d3640 1 monclient: found mon.noname-b 2024-03-29T11:29:12.379 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.383+0000 7f5b2d2d3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:12.379 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.383+0000 7f5b2d2d3640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.114:6790/0 2024-03-29T11:29:12.379 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.383+0000 7f5b2d2d3640 1 -- 172.21.15.114:0/3650761085 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b28146340 con 0x7f5b2813f650 2024-03-29T11:29:12.379 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.383+0000 7f5b2d2d3640 10 monclient: _finish_auth 0 2024-03-29T11:29:12.379 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.383+0000 7f5b2d2d3640 10 monclient: _check_auth_tickets 2024-03-29T11:29:12.379 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.383+0000 7f5b2d2d3640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:12.379 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.383+0000 7f5b2d2d3640 1 -- 172.21.15.114:0/3650761085 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f5b240015c0 con 0x7f5b2813f650 2024-03-29T11:29:12.379 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.383+0000 7f5b2d2d3640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:12.379 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.383+0000 7f5b2d2d3640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:29:12.379 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.383+0000 7f5b2d2d3640 10 monclient: dump: 2024-03-29T11:29:12.379 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:29:12.379 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:29:12.379 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:29:12.379 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:29:12.379 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:12.380 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:29:12.380 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:29:12.380 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:29:12.380 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:29:12.380 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:29:12.380 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.383+0000 7f5b2d2d3640 1 -- 172.21.15.114:0/3650761085 <== mon.2 v1:172.21.15.114:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5b24001720 con 0x7f5b2813f650 2024-03-29T11:29:12.380 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.383+0000 7f5b2d2d3640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:12.380 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.383+0000 7f5b2d2d3640 1 -- 172.21.15.114:0/3650761085 <== mon.2 v1:172.21.15.114:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f5b24001b30 con 0x7f5b2813f650 2024-03-29T11:29:12.380 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.383+0000 7f5b2d2d3640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:12.380 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.383+0000 7f5b2d2d3640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:29:12.380 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.383+0000 7f5b2d2d3640 10 monclient: dump: 2024-03-29T11:29:12.380 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:29:12.380 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:29:12.380 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:29:12.380 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:29:12.381 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:12.381 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:29:12.381 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:29:12.381 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:29:12.381 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:29:12.381 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:29:12.381 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.383+0000 7f5b30539640 5 monclient: authenticate success, global_id 4559 2024-03-29T11:29:12.381 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.383+0000 7f5b30539640 10 monclient: get_monmap_and_config success 2024-03-29T11:29:12.381 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.383+0000 7f5b30539640 4 set_mon_vals no callback set 2024-03-29T11:29:12.381 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.384+0000 7f5b30539640 10 monclient: shutdown 2024-03-29T11:29:12.381 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.384+0000 7f5b30539640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:12.381 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.384+0000 7f5b30539640 1 -- 172.21.15.114:0/3650761085 >> v1:172.21.15.114:6790/0 conn(0x7f5b2813f650 legacy=0x7f5b280b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:12.381 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.385+0000 7f5b30539640 1 -- 172.21.15.114:0/3650761085 shutdown_connections 2024-03-29T11:29:12.381 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.385+0000 7f5b30539640 1 -- 172.21.15.114:0/3650761085 >> 172.21.15.114:0/3650761085 conn(0x7f5b28066a00 msgr2=0x7f5b28066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:12.382 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.385+0000 7f5b30539640 1 -- 172.21.15.114:0/3650761085 shutdown_connections 2024-03-29T11:29:12.382 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.385+0000 7f5b30539640 1 -- 172.21.15.114:0/3650761085 wait complete. 2024-03-29T11:29:12.382 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.386+0000 7f5b30539640 10 monclient: build_initial_monmap 2024-03-29T11:29:12.382 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.386+0000 7f5b30539640 10 monclient: monmap: 2024-03-29T11:29:12.382 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:29:12.382 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:29:12.382 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T11:29:12.382 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T11:29:12.382 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:12.382 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:29:12.382 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:29:12.382 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T11:29:12.382 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T11:29:12.382 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:29:12.382 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.386+0000 7f5b30539640 1 librados: starting msgr at 2024-03-29T11:29:12.383 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.386+0000 7f5b30539640 1 librados: starting objecter 2024-03-29T11:29:12.383 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.387+0000 7f5b30539640 1 Processor -- start 2024-03-29T11:29:12.383 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.387+0000 7f5b30539640 1 -- start start 2024-03-29T11:29:12.383 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.387+0000 7f5b30539640 1 librados: setting wanted keys 2024-03-29T11:29:12.383 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.387+0000 7f5b30539640 1 librados: calling monclient init 2024-03-29T11:29:12.383 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.387+0000 7f5b30539640 10 monclient: init 2024-03-29T11:29:12.384 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.388+0000 7f5b30539640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:12.385 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.388+0000 7f5b30539640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:29:12.385 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.388+0000 7f5b30539640 10 monclient(hunting): picked mon.noname-a con 0x7f5b280a9d70 addr v1:172.21.15.114:6789/0 2024-03-29T11:29:12.385 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.388+0000 7f5b30539640 10 monclient(hunting): picked mon.noname-b con 0x7f5b2813f650 addr v1:172.21.15.176:6789/0 2024-03-29T11:29:12.385 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.389+0000 7f5b30539640 10 monclient(hunting): picked mon.noname-c con 0x7f5b281457f0 addr v1:172.21.15.114:6790/0 2024-03-29T11:29:12.385 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.389+0000 7f5b30539640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b28142e70 con 0x7f5b280a9d70 2024-03-29T11:29:12.385 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.389+0000 7f5b30539640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b28141c80 con 0x7f5b2813f650 2024-03-29T11:29:12.385 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.389+0000 7f5b30539640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b28142b00 con 0x7f5b281457f0 2024-03-29T11:29:12.385 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.389+0000 7f5b30539640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:12.385 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.389+0000 7f5b30539640 10 monclient(hunting): authenticate will time out at 1422.571592s 2024-03-29T11:29:12.385 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.393+0000 7f5b2dad4640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f5b2813f650 0x7f5b28134ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.114:43923/0 (socket says 172.21.15.114:43923) 2024-03-29T11:29:12.385 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.393+0000 7f5b2dad4640 1 -- 172.21.15.114:0/4007403773 learned_addr learned my addr 172.21.15.114:0/4007403773 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:29:12.385 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.393+0000 7f5b1effd640 10 client.?.objecter ms_handle_connect 0x7f5b2813f650 2024-03-29T11:29:12.385 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.393+0000 7f5b1effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:12.385 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.393+0000 7f5b1effd640 1 -- 172.21.15.114:0/4007403773 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3833607945 0 0) 0x7f5b28141c80 con 0x7f5b2813f650 2024-03-29T11:29:12.386 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.393+0000 7f5b1effd640 10 monclient(hunting): _negotiate 2024-03-29T11:29:12.386 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.393+0000 7f5b1effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:12.386 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.393+0000 7f5b1effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:12.386 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.393+0000 7f5b1effd640 10 monclient(hunting): my global_id is 4489 2024-03-29T11:29:12.386 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.393+0000 7f5b1effd640 1 -- 172.21.15.114:0/4007403773 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b0c004eb0 con 0x7f5b2813f650 2024-03-29T11:29:12.386 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.394+0000 7f5b1effd640 10 client.?.objecter ms_handle_connect 0x7f5b280a9d70 2024-03-29T11:29:12.386 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.394+0000 7f5b1effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:12.387 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.394+0000 7f5b1effd640 1 -- 172.21.15.114:0/4007403773 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 286276073 0 0) 0x7f5b28142e70 con 0x7f5b280a9d70 2024-03-29T11:29:12.387 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.394+0000 7f5b1effd640 10 monclient(hunting): _negotiate 2024-03-29T11:29:12.387 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.394+0000 7f5b1effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:12.387 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.394+0000 7f5b1effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:12.387 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.394+0000 7f5b1effd640 10 monclient(hunting): my global_id is 4527 2024-03-29T11:29:12.387 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.394+0000 7f5b1effd640 1 -- 172.21.15.114:0/4007403773 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b0c0051d0 con 0x7f5b280a9d70 2024-03-29T11:29:12.387 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.395+0000 7f5b1effd640 1 -- 172.21.15.114:0/4007403773 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3049944305 0 0) 0x7f5b0c0051d0 con 0x7f5b280a9d70 2024-03-29T11:29:12.387 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.395+0000 7f5b1effd640 1 -- 172.21.15.114:0/4007403773 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5b0c005640 con 0x7f5b280a9d70 2024-03-29T11:29:12.387 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.395+0000 7f5b1effd640 1 -- 172.21.15.114:0/4007403773 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f5b20001b90 con 0x7f5b280a9d70 2024-03-29T11:29:12.387 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.395+0000 7f5b1effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:12.387 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.395+0000 7f5b1effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:29:12.387 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.395+0000 7f5b1effd640 10 monclient(hunting): dump: 2024-03-29T11:29:12.387 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:29:12.388 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:29:12.388 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:29:12.388 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:29:12.388 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:12.388 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:29:12.388 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:29:12.388 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:29:12.388 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:29:12.388 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:29:12.388 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.395+0000 7f5b1effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:29:12.388 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.395+0000 7f5b1effd640 1 -- 172.21.15.114:0/4007403773 <== mon.0 v1:172.21.15.114:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2393668221 0 0) 0x7f5b0c005640 con 0x7f5b280a9d70 2024-03-29T11:29:12.388 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.395+0000 7f5b1effd640 1 -- 172.21.15.114:0/4007403773 >> v1:172.21.15.114:6790/0 conn(0x7f5b281457f0 legacy=0x7f5b281daa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:12.388 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.395+0000 7f5b1effd640 1 -- 172.21.15.114:0/4007403773 >> v1:172.21.15.176:6789/0 conn(0x7f5b2813f650 legacy=0x7f5b28134ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:12.388 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.395+0000 7f5b1effd640 10 monclient: _finish_hunting 0 2024-03-29T11:29:12.388 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.395+0000 7f5b1effd640 1 monclient: found mon.a 2024-03-29T11:29:12.389 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.395+0000 7f5b1effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:12.389 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.395+0000 7f5b1effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T11:29:12.389 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.395+0000 7f5b1effd640 1 -- 172.21.15.114:0/4007403773 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b2813e300 con 0x7f5b280a9d70 2024-03-29T11:29:12.389 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.395+0000 7f5b1effd640 10 monclient: _finish_auth 0 2024-03-29T11:29:12.389 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.395+0000 7f5b1effd640 10 monclient: _check_auth_tickets 2024-03-29T11:29:12.389 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.395+0000 7f5b1effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:12.391 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.396+0000 7f5b1effd640 1 -- 172.21.15.114:0/4007403773 <== mon.0 v1:172.21.15.114:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5b200020f0 con 0x7f5b280a9d70 2024-03-29T11:29:12.391 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.396+0000 7f5b1effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:12.391 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.396+0000 7f5b1effd640 1 -- 172.21.15.114:0/4007403773 <== mon.0 v1:172.21.15.114:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f5b20001b90 con 0x7f5b280a9d70 2024-03-29T11:29:12.391 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.396+0000 7f5b1effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:12.391 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.396+0000 7f5b1effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:29:12.391 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.396+0000 7f5b1effd640 10 monclient: dump: 2024-03-29T11:29:12.391 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:29:12.392 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:29:12.392 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:29:12.392 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:29:12.392 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:12.392 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:29:12.392 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:29:12.392 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:29:12.392 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:29:12.392 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:29:12.392 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.396+0000 7f5b2cad2640 4 set_mon_vals no callback set 2024-03-29T11:29:12.392 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.397+0000 7f5b30539640 5 monclient: authenticate success, global_id 4527 2024-03-29T11:29:12.392 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.397+0000 7f5b30539640 10 monclient: _renew_subs 2024-03-29T11:29:12.392 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.397+0000 7f5b30539640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T11:29:12.392 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.397+0000 7f5b30539640 1 -- 172.21.15.114:0/4007403773 --> v1:172.21.15.114:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b2813e5b0 con 0x7f5b280a9d70 2024-03-29T11:29:12.392 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.397+0000 7f5b30539640 10 client.4527.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:29:12.393 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.397+0000 7f5b30539640 10 monclient: _renew_subs 2024-03-29T11:29:12.393 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.397+0000 7f5b30539640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T11:29:12.393 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.397+0000 7f5b30539640 1 -- 172.21.15.114:0/4007403773 --> v1:172.21.15.114:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b28135590 con 0x7f5b280a9d70 2024-03-29T11:29:12.393 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.397+0000 7f5b30539640 1 librados: init done 2024-03-29T11:29:12.393 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.400+0000 7f5b30539640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:12.393 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.400+0000 7f5b30539640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:12.393 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.400+0000 7f5b30539640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T11:29:12.393 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.400+0000 7f5b30539640 1 -- 172.21.15.114:0/4007403773 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b28142c90 con 0x7f5b280a9d70 2024-03-29T11:29:12.394 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.400+0000 7f5b1effd640 1 -- 172.21.15.114:0/4007403773 <== mon.0 v1:172.21.15.114:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1594292461 0 0) 0x7f5b2001aa50 con 0x7f5b280a9d70 2024-03-29T11:29:12.394 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.402+0000 7f5b1effd640 1 -- 172.21.15.114:0/4007403773 <== mon.0 v1:172.21.15.114:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3238987313 0 0) 0x7f5b20054740 con 0x7f5b280a9d70 2024-03-29T11:29:12.394 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.402+0000 7f5b1effd640 10 client.4527.objecter ms_dispatch 0x7f5b28000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:29:12.394 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.402+0000 7f5b1effd640 3 client.4527.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:29:12.394 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.402+0000 7f5b1effd640 3 client.4527.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:29:12.394 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.402+0000 7f5b2dad4640 10 monclient: get_auth_request con 0x7f5b0c06e310 auth_method 0 2024-03-29T11:29:12.394 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.402+0000 7f5b1effd640 20 client.4527.objecter dump_active .. 0 homeless 2024-03-29T11:29:12.395 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.402+0000 7f5b1effd640 10 client.4527.objecter ms_handle_connect 0x7f5b0c06e310 2024-03-29T11:29:12.403 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.412+0000 7f5b1effd640 1 -- 172.21.15.114:0/4007403773 <== mon.0 v1:172.21.15.114:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5b200509e0 con 0x7f5b280a9d70 2024-03-29T11:29:12.403 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.412+0000 7f5b1effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:12.404 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.412+0000 7f5b1effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:29:12.722 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.731+0000 7f5b30539640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:12.722 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.731+0000 7f5b30539640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:12.722 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.731+0000 7f5b30539640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T11:29:12.722 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.731+0000 7f5b30539640 1 -- 172.21.15.114:0/4007403773 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5b28142c90 con 0x7f5b280a9d70 2024-03-29T11:29:12.725 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.734+0000 7f5b1effd640 1 -- 172.21.15.114:0/4007403773 <== mon.0 v1:172.21.15.114:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2858224613) 0x7f5b20050b80 con 0x7f5b280a9d70 2024-03-29T11:29:12.725 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.734+0000 7f5b1effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:12.725 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.734+0000 7f5b1effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:29:12.725 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T11:29:12.726 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"b0de2b9b-c45a-44c7-b6ee-e83b39a1e108","created":"2024-03-29T11:20:49.420660+0000","modified":"2024-03-29T11:20:51.700167+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:51.700167+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a64785d3-6b11-499a-b706-cc827ab96a74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5882dadb-e7a8-4604-ba9a-418013c106c8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d7e8c409-bc66-4e4a-bbdb-420bad31388e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0682989-3595-4cd6-90a5-ab9111c278e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"67068120-e3ff-4817-bd06-44feb7479a35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09e8b-8a85-4997-bdea-75d754e6ee67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2f0d9ead-f47f-4cb3-92a4-0a5ad23cd831","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fdb6a25f-6bd8-4c65-9b9b-b460ba7673a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:29:12.731 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.738+0000 7f5b30539640 10 librados: watch_flush enter 2024-03-29T11:29:12.732 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.740+0000 7f5b30539640 10 librados: watch_flush exit 2024-03-29T11:29:12.732 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.740+0000 7f5b30539640 20 client.4527.objecter shutdown clearing up homeless session... 2024-03-29T11:29:12.732 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.740+0000 7f5b30539640 10 client.4527.objecter successfully canceled tick 2024-03-29T11:29:12.732 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.741+0000 7f5b30539640 1 -- 172.21.15.114:0/4007403773 >> v1:172.21.15.114:6816/4038755055 conn(0x7f5b0c06e310 legacy=0x7f5b0c0707d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:12.732 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.741+0000 7f5b30539640 10 monclient: shutdown 2024-03-29T11:29:12.732 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.741+0000 7f5b30539640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:12.732 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.741+0000 7f5b30539640 1 -- 172.21.15.114:0/4007403773 >> v1:172.21.15.114:6789/0 conn(0x7f5b280a9d70 legacy=0x7f5b281423f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:12.733 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.741+0000 7f5b30539640 1 -- 172.21.15.114:0/4007403773 shutdown_connections 2024-03-29T11:29:12.733 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.741+0000 7f5b30539640 1 -- 172.21.15.114:0/4007403773 >> 172.21.15.114:0/4007403773 conn(0x7f5b28066a00 msgr2=0x7f5b280b42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:12.733 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.741+0000 7f5b30539640 1 -- 172.21.15.114:0/4007403773 shutdown_connections 2024-03-29T11:29:12.733 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.741+0000 7f5b30539640 1 -- 172.21.15.114:0/4007403773 wait complete. 2024-03-29T11:29:12.733 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:12.742+0000 7f5b30539640 1 librados: shutdown 2024-03-29T11:29:12.750 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:29:19.559 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:29:19.709 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.719+0000 7f4507cd3640 10 monclient: get_monmap_and_config 2024-03-29T11:29:19.709 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.719+0000 7f4507cd3640 10 monclient: build_initial_monmap 2024-03-29T11:29:19.709 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.719+0000 7f4507cd3640 10 monclient: monmap: 2024-03-29T11:29:19.709 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:29:19.709 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:29:19.710 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:29:19.719307+0000 2024-03-29T11:29:19.710 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:29:19.719307+0000 2024-03-29T11:29:19.710 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:19.710 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:29:19.710 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:29:19.710 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T11:29:19.710 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T11:29:19.710 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:29:19.710 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.720+0000 7f4507cd3640 1 Processor -- start 2024-03-29T11:29:19.714 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.721+0000 7f4507cd3640 1 -- start start 2024-03-29T11:29:19.714 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.721+0000 7f4507cd3640 10 monclient: init 2024-03-29T11:29:19.714 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.721+0000 7f4507cd3640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:19.714 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.721+0000 7f4507cd3640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:29:19.714 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.721+0000 7f4507cd3640 10 monclient(hunting): picked mon.noname-a con 0x7f45001422b0 addr v1:172.21.15.114:6789/0 2024-03-29T11:29:19.714 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.721+0000 7f4507cd3640 10 monclient(hunting): picked mon.noname-c con 0x7f4500142690 addr v1:172.21.15.176:6789/0 2024-03-29T11:29:19.714 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.721+0000 7f4507cd3640 10 monclient(hunting): picked mon.noname-b con 0x7f45000b17d0 addr v1:172.21.15.114:6790/0 2024-03-29T11:29:19.714 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.721+0000 7f4507cd3640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4500142c80 con 0x7f45001422b0 2024-03-29T11:29:19.714 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.721+0000 7f4507cd3640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45000aae40 con 0x7f4500142690 2024-03-29T11:29:19.714 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.721+0000 7f4507cd3640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f450013fbd0 con 0x7f45000b17d0 2024-03-29T11:29:19.714 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.721+0000 7f4507cd3640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:19.714 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.721+0000 7f4507cd3640 10 monclient(hunting): authenticate will time out at 1429.902964s 2024-03-29T11:29:19.716 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.724+0000 7f450526e640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f4500142690 0x7f45000b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.114:36124/0 (socket says 172.21.15.114:36124) 2024-03-29T11:29:19.716 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.724+0000 7f450526e640 1 -- 172.21.15.114:0/2698222547 learned_addr learned my addr 172.21.15.114:0/2698222547 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:29:19.717 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.724+0000 7f4504a6d640 1 -- 172.21.15.114:0/2698222547 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3660561536 0 0) 0x7f45000aae40 con 0x7f4500142690 2024-03-29T11:29:19.717 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.724+0000 7f4504a6d640 10 monclient(hunting): _negotiate 2024-03-29T11:29:19.717 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.724+0000 7f4504a6d640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:19.717 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.724+0000 7f4504a6d640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:19.717 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.724+0000 7f4504a6d640 10 monclient(hunting): my global_id is 4492 2024-03-29T11:29:19.717 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.724+0000 7f4504a6d640 1 -- 172.21.15.114:0/2698222547 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44ec003520 con 0x7f4500142690 2024-03-29T11:29:19.717 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.725+0000 7f4504a6d640 1 -- 172.21.15.114:0/2698222547 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 251113752 0 0) 0x7f44ec003520 con 0x7f4500142690 2024-03-29T11:29:19.717 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.725+0000 7f4504a6d640 1 -- 172.21.15.114:0/2698222547 >> v1:172.21.15.114:6790/0 conn(0x7f45000b17d0 legacy=0x7f45000b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:29:19.717 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.725+0000 7f4504a6d640 1 -- 172.21.15.114:0/2698222547 >> v1:172.21.15.114:6789/0 conn(0x7f45001422b0 legacy=0x7f450013f4c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:29:19.720 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.726+0000 7f4504a6d640 10 monclient: _finish_hunting 0 2024-03-29T11:29:19.720 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.726+0000 7f4504a6d640 1 monclient: found mon.noname-c 2024-03-29T11:29:19.720 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.726+0000 7f4504a6d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:19.720 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.726+0000 7f4504a6d640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.176:6789/0 2024-03-29T11:29:19.720 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.727+0000 7f4504a6d640 1 -- 172.21.15.114:0/2698222547 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4500151940 con 0x7f4500142690 2024-03-29T11:29:19.720 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.727+0000 7f4504a6d640 10 monclient: _finish_auth 0 2024-03-29T11:29:19.720 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.727+0000 7f4504a6d640 10 monclient: _check_auth_tickets 2024-03-29T11:29:19.720 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.727+0000 7f4504a6d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:19.720 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.727+0000 7f4504a6d640 1 -- 172.21.15.114:0/2698222547 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f44f80019f0 con 0x7f4500142690 2024-03-29T11:29:19.720 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.727+0000 7f4504a6d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:19.720 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.727+0000 7f4504a6d640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:29:19.721 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.727+0000 7f4504a6d640 10 monclient: dump: 2024-03-29T11:29:19.721 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:29:19.721 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:29:19.721 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:29:19.721 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:29:19.721 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:19.721 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:29:19.721 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:29:19.721 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:29:19.721 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:29:19.721 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:29:19.721 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.728+0000 7f4507cd3640 5 monclient: authenticate success, global_id 4492 2024-03-29T11:29:19.721 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.728+0000 7f4507cd3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:19.721 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.728+0000 7f4504a6d640 1 -- 172.21.15.114:0/2698222547 <== mon.1 v1:172.21.15.176:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f44f8001c50 con 0x7f4500142690 2024-03-29T11:29:19.722 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.728+0000 7f4504a6d640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:19.722 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.728+0000 7f4504a6d640 1 -- 172.21.15.114:0/2698222547 <== mon.1 v1:172.21.15.176:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f44f8002020 con 0x7f4500142690 2024-03-29T11:29:19.722 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.728+0000 7f4504a6d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:19.722 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.728+0000 7f4504a6d640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:29:19.722 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.728+0000 7f4504a6d640 10 monclient: dump: 2024-03-29T11:29:19.722 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:29:19.722 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:29:19.722 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:29:19.722 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:29:19.722 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:19.722 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:29:19.722 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:29:19.722 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:29:19.722 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:29:19.722 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:29:19.723 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.728+0000 7f4507cd3640 10 monclient: get_monmap_and_config success 2024-03-29T11:29:19.723 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.728+0000 7f4507cd3640 4 set_mon_vals no callback set 2024-03-29T11:29:19.723 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.729+0000 7f4507cd3640 10 monclient: shutdown 2024-03-29T11:29:19.723 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.729+0000 7f4507cd3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:19.723 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.729+0000 7f4507cd3640 1 -- 172.21.15.114:0/2698222547 >> v1:172.21.15.176:6789/0 conn(0x7f4500142690 legacy=0x7f45000b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:19.723 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.729+0000 7f4507cd3640 1 -- 172.21.15.114:0/2698222547 shutdown_connections 2024-03-29T11:29:19.723 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.729+0000 7f4507cd3640 1 -- 172.21.15.114:0/2698222547 >> 172.21.15.114:0/2698222547 conn(0x7f4500066a00 msgr2=0x7f4500066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:19.728 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.731+0000 7f4507cd3640 1 -- 172.21.15.114:0/2698222547 shutdown_connections 2024-03-29T11:29:19.728 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.733+0000 7f4507cd3640 1 -- 172.21.15.114:0/2698222547 wait complete. 2024-03-29T11:29:19.728 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.734+0000 7f4507cd3640 10 monclient: build_initial_monmap 2024-03-29T11:29:19.728 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.734+0000 7f4507cd3640 10 monclient: monmap: 2024-03-29T11:29:19.728 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:29:19.728 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:29:19.729 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T11:29:19.729 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T11:29:19.729 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:19.729 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:29:19.729 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:29:19.730 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T11:29:19.730 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T11:29:19.730 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:29:19.730 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.734+0000 7f4507cd3640 1 librados: starting msgr at 2024-03-29T11:29:19.730 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.734+0000 7f4507cd3640 1 librados: starting objecter 2024-03-29T11:29:19.730 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.734+0000 7f4507cd3640 1 Processor -- start 2024-03-29T11:29:19.731 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.734+0000 7f4507cd3640 1 -- start start 2024-03-29T11:29:19.731 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.734+0000 7f4507cd3640 1 librados: setting wanted keys 2024-03-29T11:29:19.731 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.734+0000 7f4507cd3640 1 librados: calling monclient init 2024-03-29T11:29:19.731 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.734+0000 7f4507cd3640 10 monclient: init 2024-03-29T11:29:19.731 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.735+0000 7f4507cd3640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:19.731 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.735+0000 7f4507cd3640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:29:19.731 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.735+0000 7f4507cd3640 10 monclient(hunting): picked mon.noname-b con 0x7f45000b17d0 addr v1:172.21.15.176:6789/0 2024-03-29T11:29:19.732 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.735+0000 7f4507cd3640 10 monclient(hunting): picked mon.noname-a con 0x7f45001422b0 addr v1:172.21.15.114:6789/0 2024-03-29T11:29:19.732 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.735+0000 7f4507cd3640 10 monclient(hunting): picked mon.noname-c con 0x7f450013bda0 addr v1:172.21.15.114:6790/0 2024-03-29T11:29:19.732 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.735+0000 7f4507cd3640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f450012ccb0 con 0x7f45001422b0 2024-03-29T11:29:19.732 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.735+0000 7f4507cd3640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f450012d140 con 0x7f45000b17d0 2024-03-29T11:29:19.732 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.735+0000 7f4507cd3640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f450012c650 con 0x7f450013bda0 2024-03-29T11:29:19.732 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.735+0000 7f4507cd3640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:19.733 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.735+0000 7f4507cd3640 10 monclient(hunting): authenticate will time out at 1429.916808s 2024-03-29T11:29:19.742 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.738+0000 7f4506270640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f450013bda0 0x7f450012bf40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:60181/0 (socket says 172.21.15.114:60181) 2024-03-29T11:29:19.743 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.738+0000 7f4506270640 1 -- 172.21.15.114:0/1434904867 learned_addr learned my addr 172.21.15.114:0/1434904867 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:29:19.743 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.738+0000 7f44f67fc640 10 client.?.objecter ms_handle_connect 0x7f450013bda0 2024-03-29T11:29:19.743 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.738+0000 7f44f67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:19.743 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.739+0000 7f44f67fc640 10 client.?.objecter ms_handle_connect 0x7f45000b17d0 2024-03-29T11:29:19.743 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.739+0000 7f44f67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:19.743 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.739+0000 7f44f67fc640 1 -- 172.21.15.114:0/1434904867 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1010831603 0 0) 0x7f450012c650 con 0x7f450013bda0 2024-03-29T11:29:19.743 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.739+0000 7f44f67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:29:19.743 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.739+0000 7f44f67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:19.743 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.739+0000 7f44f67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:19.743 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.739+0000 7f44f67fc640 10 monclient(hunting): my global_id is 4562 2024-03-29T11:29:19.743 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.739+0000 7f44f67fc640 1 -- 172.21.15.114:0/1434904867 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44e4004eb0 con 0x7f450013bda0 2024-03-29T11:29:19.743 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.739+0000 7f44f67fc640 1 -- 172.21.15.114:0/1434904867 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3728307008 0 0) 0x7f450012d140 con 0x7f45000b17d0 2024-03-29T11:29:19.743 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.739+0000 7f44f67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:29:19.743 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.739+0000 7f44f67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:19.744 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.739+0000 7f44f67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:19.744 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.739+0000 7f44f67fc640 10 monclient(hunting): my global_id is 4495 2024-03-29T11:29:19.744 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.739+0000 7f44f67fc640 1 -- 172.21.15.114:0/1434904867 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f450012c650 con 0x7f45000b17d0 2024-03-29T11:29:19.744 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.740+0000 7f44f67fc640 1 -- 172.21.15.114:0/1434904867 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3301763415 0 0) 0x7f44e4004eb0 con 0x7f450013bda0 2024-03-29T11:29:19.744 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.740+0000 7f44f67fc640 1 -- 172.21.15.114:0/1434904867 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f450012d140 con 0x7f450013bda0 2024-03-29T11:29:19.744 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.740+0000 7f44f67fc640 1 -- 172.21.15.114:0/1434904867 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3451290774 0 0) 0x7f450012c650 con 0x7f45000b17d0 2024-03-29T11:29:19.744 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.740+0000 7f44f67fc640 1 -- 172.21.15.114:0/1434904867 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f44e4004eb0 con 0x7f45000b17d0 2024-03-29T11:29:19.744 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.740+0000 7f44f67fc640 1 -- 172.21.15.114:0/1434904867 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f44fc0018e0 con 0x7f450013bda0 2024-03-29T11:29:19.744 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.740+0000 7f44f67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:19.744 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.740+0000 7f44f67fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:29:19.744 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.740+0000 7f44f67fc640 10 monclient(hunting): dump: 2024-03-29T11:29:19.744 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:29:19.744 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:29:19.744 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:29:19.745 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:29:19.745 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:19.745 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:29:19.745 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:29:19.745 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:29:19.745 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:29:19.745 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:29:19.745 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.740+0000 7f44f67fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:29:19.745 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.740+0000 7f44f67fc640 1 -- 172.21.15.114:0/1434904867 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f44f0001710 con 0x7f45000b17d0 2024-03-29T11:29:19.745 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.740+0000 7f44f67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:19.745 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.740+0000 7f44f67fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:29:19.745 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.740+0000 7f44f67fc640 10 monclient(hunting): dump: 2024-03-29T11:29:19.745 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:29:19.745 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:29:19.745 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:29:19.746 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:29:19.746 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:19.746 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:29:19.746 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:29:19.746 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:29:19.746 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:29:19.746 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:29:19.746 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.741+0000 7f44f67fc640 1 -- 172.21.15.114:0/1434904867 <== mon.1 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3720405093 0 0) 0x7f44e4004eb0 con 0x7f45000b17d0 2024-03-29T11:29:19.746 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.741+0000 7f44f67fc640 1 -- 172.21.15.114:0/1434904867 >> v1:172.21.15.114:6790/0 conn(0x7f450013bda0 legacy=0x7f450012bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:19.746 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.741+0000 7f44f67fc640 1 -- 172.21.15.114:0/1434904867 >> v1:172.21.15.114:6789/0 conn(0x7f45001422b0 legacy=0x7f450012b830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:19.746 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.741+0000 7f44f67fc640 10 monclient: _finish_hunting 0 2024-03-29T11:29:19.746 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.741+0000 7f44f67fc640 1 monclient: found mon.b 2024-03-29T11:29:19.746 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.741+0000 7f44f67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:19.746 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.741+0000 7f44f67fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:29:19.747 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.741+0000 7f44f67fc640 1 -- 172.21.15.114:0/1434904867 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f450012c940 con 0x7f45000b17d0 2024-03-29T11:29:19.747 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.741+0000 7f44f67fc640 10 monclient: _finish_auth 0 2024-03-29T11:29:19.747 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.741+0000 7f44f67fc640 10 monclient: _check_auth_tickets 2024-03-29T11:29:19.747 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.741+0000 7f44f67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:19.747 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.741+0000 7f4507cd3640 5 monclient: authenticate success, global_id 4495 2024-03-29T11:29:19.747 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.741+0000 7f4507cd3640 10 monclient: _renew_subs 2024-03-29T11:29:19.747 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.741+0000 7f4507cd3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:29:19.747 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.741+0000 7f4507cd3640 1 -- 172.21.15.114:0/1434904867 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4500131aa0 con 0x7f45000b17d0 2024-03-29T11:29:19.747 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.741+0000 7f4507cd3640 10 client.4495.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:29:19.747 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.741+0000 7f4507cd3640 10 monclient: _renew_subs 2024-03-29T11:29:19.747 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.741+0000 7f4507cd3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:29:19.747 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.741+0000 7f4507cd3640 1 -- 172.21.15.114:0/1434904867 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f45001dfbd0 con 0x7f45000b17d0 2024-03-29T11:29:19.747 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.741+0000 7f4507cd3640 1 librados: init done 2024-03-29T11:29:19.747 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.741+0000 7f44f67fc640 1 -- 172.21.15.114:0/1434904867 <== mon.1 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f44f0001870 con 0x7f45000b17d0 2024-03-29T11:29:19.747 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.741+0000 7f44f67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:19.748 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.741+0000 7f44f7fff640 4 set_mon_vals no callback set 2024-03-29T11:29:19.748 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.741+0000 7f44f67fc640 1 -- 172.21.15.114:0/1434904867 <== mon.1 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f44f0003790 con 0x7f45000b17d0 2024-03-29T11:29:19.748 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.742+0000 7f44f67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:19.748 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.742+0000 7f44f67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:29:19.748 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.742+0000 7f44f67fc640 10 monclient: dump: 2024-03-29T11:29:19.748 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:29:19.748 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:29:19.748 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:29:19.748 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:29:19.748 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:19.748 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:29:19.748 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:29:19.748 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:29:19.748 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:29:19.748 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:29:19.749 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.743+0000 7f4507cd3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:19.749 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.743+0000 7f4507cd3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:19.749 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.743+0000 7f4507cd3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:29:19.749 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.743+0000 7f4507cd3640 1 -- 172.21.15.114:0/1434904867 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44e8005080 con 0x7f45000b17d0 2024-03-29T11:29:19.749 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.743+0000 7f44f67fc640 1 -- 172.21.15.114:0/1434904867 <== mon.1 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1594292461 0 0) 0x7f44f0019760 con 0x7f45000b17d0 2024-03-29T11:29:19.749 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.744+0000 7f44f67fc640 1 -- 172.21.15.114:0/1434904867 <== mon.1 v1:172.21.15.176:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3238987313 0 0) 0x7f44f00836f0 con 0x7f45000b17d0 2024-03-29T11:29:19.749 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.744+0000 7f450526e640 10 monclient: get_auth_request con 0x7f44e406df00 auth_method 0 2024-03-29T11:29:19.749 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.745+0000 7f44f67fc640 10 client.4495.objecter ms_dispatch 0x7f4500000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:29:19.749 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.745+0000 7f44f67fc640 3 client.4495.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:29:19.750 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.745+0000 7f44f67fc640 3 client.4495.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:29:19.750 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.745+0000 7f44f67fc640 20 client.4495.objecter dump_active .. 0 homeless 2024-03-29T11:29:19.750 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.745+0000 7f44f67fc640 10 client.4495.objecter ms_handle_connect 0x7f44e406df00 2024-03-29T11:29:19.750 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.750+0000 7f44f67fc640 1 -- 172.21.15.114:0/1434904867 <== mon.1 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f44f004efd0 con 0x7f45000b17d0 2024-03-29T11:29:19.750 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.751+0000 7f44f67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:19.750 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:19.751+0000 7f44f67fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:29:20.055 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:20.064+0000 7f4507cd3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:20.055 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:20.064+0000 7f4507cd3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:20.056 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:20.064+0000 7f4507cd3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:29:20.056 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:20.064+0000 7f4507cd3640 1 -- 172.21.15.114:0/1434904867 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f44e8005330 con 0x7f45000b17d0 2024-03-29T11:29:20.056 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:20.065+0000 7f44f67fc640 1 -- 172.21.15.114:0/1434904867 <== mon.1 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2858224613) 0x7f44f0050f00 con 0x7f45000b17d0 2024-03-29T11:29:20.056 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:20.065+0000 7f44f67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:20.057 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T11:29:20.057 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"b0de2b9b-c45a-44c7-b6ee-e83b39a1e108","created":"2024-03-29T11:20:49.420660+0000","modified":"2024-03-29T11:20:51.700167+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:51.700167+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a64785d3-6b11-499a-b706-cc827ab96a74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5882dadb-e7a8-4604-ba9a-418013c106c8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d7e8c409-bc66-4e4a-bbdb-420bad31388e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0682989-3595-4cd6-90a5-ab9111c278e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"67068120-e3ff-4817-bd06-44feb7479a35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09e8b-8a85-4997-bdea-75d754e6ee67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2f0d9ead-f47f-4cb3-92a4-0a5ad23cd831","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fdb6a25f-6bd8-4c65-9b9b-b460ba7673a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:29:20.057 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:20.065+0000 7f44f67fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:29:20.059 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:20.068+0000 7f4507cd3640 10 librados: watch_flush enter 2024-03-29T11:29:20.059 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:20.068+0000 7f4507cd3640 10 librados: watch_flush exit 2024-03-29T11:29:20.059 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:20.068+0000 7f4507cd3640 20 client.4495.objecter shutdown clearing up homeless session... 2024-03-29T11:29:20.060 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:20.068+0000 7f4507cd3640 10 client.4495.objecter successfully canceled tick 2024-03-29T11:29:20.060 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:20.069+0000 7f4507cd3640 1 -- 172.21.15.114:0/1434904867 >> v1:172.21.15.114:6816/4038755055 conn(0x7f44e406df00 legacy=0x7f44e40703c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:20.060 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:20.069+0000 7f4507cd3640 10 monclient: shutdown 2024-03-29T11:29:20.060 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:20.069+0000 7f4507cd3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:20.061 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:20.069+0000 7f4507cd3640 1 -- 172.21.15.114:0/1434904867 >> v1:172.21.15.176:6789/0 conn(0x7f45000b17d0 legacy=0x7f45001df2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:20.061 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:20.069+0000 7f4507cd3640 1 -- 172.21.15.114:0/1434904867 shutdown_connections 2024-03-29T11:29:20.061 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:20.069+0000 7f4507cd3640 1 -- 172.21.15.114:0/1434904867 >> 172.21.15.114:0/1434904867 conn(0x7f4500066a00 msgr2=0x7f450013dbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:20.061 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:20.069+0000 7f4507cd3640 1 -- 172.21.15.114:0/1434904867 shutdown_connections 2024-03-29T11:29:20.061 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:20.069+0000 7f4507cd3640 1 -- 172.21.15.114:0/1434904867 wait complete. 2024-03-29T11:29:20.061 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:20.069+0000 7f4507cd3640 1 librados: shutdown 2024-03-29T11:29:20.077 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:29:26.884 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:29:27.040 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.048+0000 7f7bfb217640 10 monclient: get_monmap_and_config 2024-03-29T11:29:27.040 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.048+0000 7f7bfb217640 10 monclient: build_initial_monmap 2024-03-29T11:29:27.040 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.048+0000 7f7bfb217640 10 monclient: monmap: 2024-03-29T11:29:27.040 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:29:27.040 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:29:27.040 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:29:27.049933+0000 2024-03-29T11:29:27.040 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:29:27.049933+0000 2024-03-29T11:29:27.040 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:27.041 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:29:27.041 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:29:27.041 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T11:29:27.041 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T11:29:27.041 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:29:27.041 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.050+0000 7f7bfb217640 1 Processor -- start 2024-03-29T11:29:27.043 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.051+0000 7f7bfb217640 1 -- start start 2024-03-29T11:29:27.043 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.051+0000 7f7bfb217640 10 monclient: init 2024-03-29T11:29:27.043 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.052+0000 7f7bfb217640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:27.043 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.052+0000 7f7bfb217640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:29:27.043 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.052+0000 7f7bfb217640 10 monclient(hunting): picked mon.noname-c con 0x7f7bf41422b0 addr v1:172.21.15.176:6789/0 2024-03-29T11:29:27.043 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.052+0000 7f7bfb217640 10 monclient(hunting): picked mon.noname-a con 0x7f7bf4142690 addr v1:172.21.15.114:6789/0 2024-03-29T11:29:27.043 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.052+0000 7f7bfb217640 10 monclient(hunting): picked mon.noname-b con 0x7f7bf40b17d0 addr v1:172.21.15.114:6790/0 2024-03-29T11:29:27.044 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.052+0000 7f7bfb217640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bf4142c80 con 0x7f7bf4142690 2024-03-29T11:29:27.044 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.052+0000 7f7bfb217640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bf40aae40 con 0x7f7bf41422b0 2024-03-29T11:29:27.044 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.052+0000 7f7bfb217640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bf413fbd0 con 0x7f7bf40b17d0 2024-03-29T11:29:27.044 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.052+0000 7f7bfb217640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:27.044 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.052+0000 7f7bfb217640 10 monclient(hunting): authenticate will time out at 1437.234396s 2024-03-29T11:29:27.045 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.053+0000 7f7bf8fb3640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f7bf41422b0 0x7f7bf413f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.114:35906/0 (socket says 172.21.15.114:35906) 2024-03-29T11:29:27.045 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.054+0000 7f7bf8fb3640 1 -- 172.21.15.114:0/2501627859 learned_addr learned my addr 172.21.15.114:0/2501627859 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:29:27.046 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.054+0000 7f7bf37fe640 1 -- 172.21.15.114:0/2501627859 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 729773737 0 0) 0x7f7bf40aae40 con 0x7f7bf41422b0 2024-03-29T11:29:27.046 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.054+0000 7f7bf37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:29:27.046 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.054+0000 7f7bf37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:27.046 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.054+0000 7f7bf37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:27.046 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.054+0000 7f7bf37fe640 10 monclient(hunting): my global_id is 4498 2024-03-29T11:29:27.047 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.055+0000 7f7bf37fe640 1 -- 172.21.15.114:0/2501627859 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7bdc003520 con 0x7f7bf41422b0 2024-03-29T11:29:27.048 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.056+0000 7f7bf37fe640 1 -- 172.21.15.114:0/2501627859 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2713607265 0 0) 0x7f7bdc003520 con 0x7f7bf41422b0 2024-03-29T11:29:27.048 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.056+0000 7f7bf37fe640 1 -- 172.21.15.114:0/2501627859 >> v1:172.21.15.114:6790/0 conn(0x7f7bf40b17d0 legacy=0x7f7bf40b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:29:27.048 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.056+0000 7f7bf37fe640 1 -- 172.21.15.114:0/2501627859 >> v1:172.21.15.114:6789/0 conn(0x7f7bf4142690 legacy=0x7f7bf40b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:29:27.048 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.056+0000 7f7bf37fe640 10 monclient: _finish_hunting 0 2024-03-29T11:29:27.048 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.056+0000 7f7bf37fe640 1 monclient: found mon.noname-c 2024-03-29T11:29:27.048 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.056+0000 7f7bf37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:27.048 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.056+0000 7f7bf37fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.176:6789/0 2024-03-29T11:29:27.049 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.056+0000 7f7bf37fe640 1 -- 172.21.15.114:0/2501627859 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bf4151940 con 0x7f7bf41422b0 2024-03-29T11:29:27.049 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.056+0000 7f7bf37fe640 10 monclient: _finish_auth 0 2024-03-29T11:29:27.049 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.056+0000 7f7bf37fe640 10 monclient: _check_auth_tickets 2024-03-29T11:29:27.049 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.056+0000 7f7bf37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:27.049 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.056+0000 7f7bfb217640 5 monclient: authenticate success, global_id 4498 2024-03-29T11:29:27.049 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.056+0000 7f7bfb217640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:27.049 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.056+0000 7f7bf37fe640 1 -- 172.21.15.114:0/2501627859 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f7be4001690 con 0x7f7bf41422b0 2024-03-29T11:29:27.049 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.056+0000 7f7bf37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:27.049 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.056+0000 7f7bf37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:29:27.049 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.056+0000 7f7bf37fe640 10 monclient: dump: 2024-03-29T11:29:27.049 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:29:27.049 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:29:27.049 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:29:27.049 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:29:27.049 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:27.050 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:29:27.050 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:29:27.050 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:29:27.050 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:29:27.050 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:29:27.050 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.056+0000 7f7bfb217640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:27.051 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.058+0000 7f7bf37fe640 1 -- 172.21.15.114:0/2501627859 <== mon.1 v1:172.21.15.176:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7be40018b0 con 0x7f7bf41422b0 2024-03-29T11:29:27.051 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.058+0000 7f7bf37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:27.051 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.058+0000 7f7bfb217640 10 monclient: get_monmap_and_config success 2024-03-29T11:29:27.051 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.058+0000 7f7bfb217640 4 set_mon_vals no callback set 2024-03-29T11:29:27.051 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.058+0000 7f7bf37fe640 1 -- 172.21.15.114:0/2501627859 <== mon.1 v1:172.21.15.176:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f7be4001c40 con 0x7f7bf41422b0 2024-03-29T11:29:27.051 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.058+0000 7f7bf37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:27.051 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.058+0000 7f7bf37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:29:27.051 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.058+0000 7f7bf37fe640 10 monclient: dump: 2024-03-29T11:29:27.051 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:29:27.052 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:29:27.052 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:29:27.052 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:29:27.052 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:27.052 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:29:27.052 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:29:27.052 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:29:27.052 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:29:27.052 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:29:27.052 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.059+0000 7f7bfb217640 10 monclient: shutdown 2024-03-29T11:29:27.052 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.059+0000 7f7bfb217640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:27.052 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.059+0000 7f7bfb217640 1 -- 172.21.15.114:0/2501627859 >> v1:172.21.15.176:6789/0 conn(0x7f7bf41422b0 legacy=0x7f7bf413f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:27.052 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.059+0000 7f7bfb217640 1 -- 172.21.15.114:0/2501627859 shutdown_connections 2024-03-29T11:29:27.052 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.059+0000 7f7bfb217640 1 -- 172.21.15.114:0/2501627859 >> 172.21.15.114:0/2501627859 conn(0x7f7bf4066a00 msgr2=0x7f7bf4066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:27.055 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.062+0000 7f7bfb217640 1 -- 172.21.15.114:0/2501627859 shutdown_connections 2024-03-29T11:29:27.056 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.062+0000 7f7bfb217640 1 -- 172.21.15.114:0/2501627859 wait complete. 2024-03-29T11:29:27.056 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.063+0000 7f7bfb217640 10 monclient: build_initial_monmap 2024-03-29T11:29:27.056 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.063+0000 7f7bfb217640 10 monclient: monmap: 2024-03-29T11:29:27.056 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:29:27.056 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:29:27.056 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T11:29:27.056 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T11:29:27.056 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:27.056 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:29:27.056 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:29:27.056 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T11:29:27.056 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T11:29:27.056 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:29:27.056 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.064+0000 7f7bfb217640 1 librados: starting msgr at 2024-03-29T11:29:27.057 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.064+0000 7f7bfb217640 1 librados: starting objecter 2024-03-29T11:29:27.057 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.064+0000 7f7bfb217640 1 Processor -- start 2024-03-29T11:29:27.057 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.064+0000 7f7bfb217640 1 -- start start 2024-03-29T11:29:27.057 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.064+0000 7f7bfb217640 1 librados: setting wanted keys 2024-03-29T11:29:27.057 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.064+0000 7f7bfb217640 1 librados: calling monclient init 2024-03-29T11:29:27.057 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.064+0000 7f7bfb217640 10 monclient: init 2024-03-29T11:29:27.057 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.066+0000 7f7bfb217640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:27.057 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.066+0000 7f7bfb217640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:29:27.065 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.067+0000 7f7bfb217640 10 monclient(hunting): picked mon.noname-c con 0x7f7bf40b17d0 addr v1:172.21.15.114:6790/0 2024-03-29T11:29:27.065 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.068+0000 7f7bfb217640 10 monclient(hunting): picked mon.noname-a con 0x7f7bf41422b0 addr v1:172.21.15.114:6789/0 2024-03-29T11:29:27.065 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.068+0000 7f7bfb217640 10 monclient(hunting): picked mon.noname-b con 0x7f7bf4142690 addr v1:172.21.15.176:6789/0 2024-03-29T11:29:27.065 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.068+0000 7f7bfb217640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bf40ab540 con 0x7f7bf41422b0 2024-03-29T11:29:27.066 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.068+0000 7f7bfb217640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bf40acba0 con 0x7f7bf4142690 2024-03-29T11:29:27.066 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.068+0000 7f7bfb217640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bf414e320 con 0x7f7bf40b17d0 2024-03-29T11:29:27.066 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.068+0000 7f7bfb217640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:27.066 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.068+0000 7f7bfb217640 10 monclient(hunting): authenticate will time out at 1437.250301s 2024-03-29T11:29:27.066 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.070+0000 7f7bf97b4640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f7bf4142690 0x7f7bf414dc10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.114:36494/0 (socket says 172.21.15.114:36494) 2024-03-29T11:29:27.066 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.070+0000 7f7bf97b4640 1 -- 172.21.15.114:0/2458761811 learned_addr learned my addr 172.21.15.114:0/2458761811 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:29:27.066 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.070+0000 7f7bf17fa640 10 client.?.objecter ms_handle_connect 0x7f7bf4142690 2024-03-29T11:29:27.066 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.070+0000 7f7bf17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:27.066 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.070+0000 7f7bf17fa640 1 -- 172.21.15.114:0/2458761811 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4034197482 0 0) 0x7f7bf40acba0 con 0x7f7bf4142690 2024-03-29T11:29:27.066 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.070+0000 7f7bf17fa640 10 monclient(hunting): _negotiate 2024-03-29T11:29:27.066 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.070+0000 7f7bf17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:27.066 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.070+0000 7f7bf17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:27.066 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.070+0000 7f7bf17fa640 10 monclient(hunting): my global_id is 4501 2024-03-29T11:29:27.066 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.070+0000 7f7bf17fa640 1 -- 172.21.15.114:0/2458761811 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7be0004eb0 con 0x7f7bf4142690 2024-03-29T11:29:27.067 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.070+0000 7f7bf17fa640 1 -- 172.21.15.114:0/2458761811 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 549973071 0 0) 0x7f7be0004eb0 con 0x7f7bf4142690 2024-03-29T11:29:27.067 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.070+0000 7f7bf17fa640 1 -- 172.21.15.114:0/2458761811 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7bf40acba0 con 0x7f7bf4142690 2024-03-29T11:29:27.067 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.070+0000 7f7bf17fa640 1 -- 172.21.15.114:0/2458761811 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f7bec001c60 con 0x7f7bf4142690 2024-03-29T11:29:27.067 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.070+0000 7f7bf17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:27.067 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.070+0000 7f7bf17fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:29:27.067 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.070+0000 7f7bf17fa640 10 monclient(hunting): dump: 2024-03-29T11:29:27.067 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:29:27.067 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:29:27.067 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:29:27.067 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:29:27.067 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:27.067 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:29:27.067 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:29:27.067 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:29:27.067 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:29:27.068 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:29:27.068 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.070+0000 7f7bf17fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:29:27.068 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.071+0000 7f7bf17fa640 1 -- 172.21.15.114:0/2458761811 <== mon.1 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3852243103 0 0) 0x7f7bf40acba0 con 0x7f7bf4142690 2024-03-29T11:29:27.068 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.071+0000 7f7bf17fa640 1 -- 172.21.15.114:0/2458761811 >> v1:172.21.15.114:6790/0 conn(0x7f7bf40b17d0 legacy=0x7f7bf40b56d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:27.068 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.071+0000 7f7bf17fa640 1 -- 172.21.15.114:0/2458761811 >> v1:172.21.15.114:6789/0 conn(0x7f7bf41422b0 legacy=0x7f7bf414d500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:29:27.068 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.071+0000 7f7bf17fa640 10 monclient: _finish_hunting 0 2024-03-29T11:29:27.068 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.071+0000 7f7bf17fa640 1 monclient: found mon.b 2024-03-29T11:29:27.068 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.071+0000 7f7bf17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:27.068 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.071+0000 7f7bf17fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:29:27.068 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.071+0000 7f7bf17fa640 1 -- 172.21.15.114:0/2458761811 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bf413eef0 con 0x7f7bf4142690 2024-03-29T11:29:27.069 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.071+0000 7f7bf17fa640 10 monclient: _finish_auth 0 2024-03-29T11:29:27.069 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.071+0000 7f7bf17fa640 10 monclient: _check_auth_tickets 2024-03-29T11:29:27.069 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.071+0000 7f7bf17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:27.069 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.071+0000 7f7bfb217640 5 monclient: authenticate success, global_id 4501 2024-03-29T11:29:27.069 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.071+0000 7f7bfb217640 10 monclient: _renew_subs 2024-03-29T11:29:27.069 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.071+0000 7f7bfb217640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:29:27.069 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.071+0000 7f7bfb217640 1 -- 172.21.15.114:0/2458761811 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bf40ace10 con 0x7f7bf4142690 2024-03-29T11:29:27.069 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.072+0000 7f7bfb217640 10 client.4501.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:29:27.069 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.072+0000 7f7bfb217640 10 monclient: _renew_subs 2024-03-29T11:29:27.069 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.072+0000 7f7bfb217640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:29:27.070 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.072+0000 7f7bfb217640 1 -- 172.21.15.114:0/2458761811 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bf40ab9d0 con 0x7f7bf4142690 2024-03-29T11:29:27.070 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.072+0000 7f7bfb217640 1 librados: init done 2024-03-29T11:29:27.070 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.072+0000 7f7bf17fa640 1 -- 172.21.15.114:0/2458761811 <== mon.1 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7bec003970 con 0x7f7bf4142690 2024-03-29T11:29:27.070 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.072+0000 7f7bf17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:27.070 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.072+0000 7f7bf17fa640 1 -- 172.21.15.114:0/2458761811 <== mon.1 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f7bec003d80 con 0x7f7bf4142690 2024-03-29T11:29:27.070 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.072+0000 7f7bf17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:27.070 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.072+0000 7f7bf17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:29:27.070 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.072+0000 7f7bf17fa640 10 monclient: dump: 2024-03-29T11:29:27.070 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:29:27.070 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:29:27.070 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:29:27.070 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:29:27.070 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:27.070 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:29:27.071 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:29:27.071 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:29:27.071 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:29:27.071 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:29:27.072 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.074+0000 7f7bf17fa640 1 -- 172.21.15.114:0/2458761811 <== mon.1 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1594292461 0 0) 0x7f7bec003ee0 con 0x7f7bf4142690 2024-03-29T11:29:27.072 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.073+0000 7f7bf2ffd640 4 set_mon_vals no callback set 2024-03-29T11:29:27.072 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.076+0000 7f7bfb217640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:27.072 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.076+0000 7f7bfb217640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:27.073 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.076+0000 7f7bfb217640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:29:27.073 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.076+0000 7f7bf17fa640 1 -- 172.21.15.114:0/2458761811 <== mon.1 v1:172.21.15.176:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3238987313 0 0) 0x7f7bec084df0 con 0x7f7bf4142690 2024-03-29T11:29:27.073 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.076+0000 7f7bf17fa640 10 client.4501.objecter ms_dispatch 0x7f7bf4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:29:27.073 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.076+0000 7f7bf17fa640 3 client.4501.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:29:27.073 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.076+0000 7f7bf17fa640 3 client.4501.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:29:27.073 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.076+0000 7f7bfb217640 1 -- 172.21.15.114:0/2458761811 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bf4075f10 con 0x7f7bf4142690 2024-03-29T11:29:27.073 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.076+0000 7f7bf17fa640 20 client.4501.objecter dump_active .. 0 homeless 2024-03-29T11:29:27.073 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.076+0000 7f7bf8fb3640 10 monclient: get_auth_request con 0x7f7be006dc90 auth_method 0 2024-03-29T11:29:27.073 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.077+0000 7f7bf17fa640 10 client.4501.objecter ms_handle_connect 0x7f7be006dc90 2024-03-29T11:29:27.076 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.084+0000 7f7bf17fa640 1 -- 172.21.15.114:0/2458761811 <== mon.1 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7bec0506d0 con 0x7f7bf4142690 2024-03-29T11:29:27.076 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.084+0000 7f7bf17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:27.076 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.084+0000 7f7bf17fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:29:27.442 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.451+0000 7f7bfb217640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:27.442 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.451+0000 7f7bfb217640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:27.443 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.451+0000 7f7bfb217640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:29:27.443 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.451+0000 7f7bfb217640 1 -- 172.21.15.114:0/2458761811 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7bf40b5490 con 0x7f7bf4142690 2024-03-29T11:29:27.443 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.452+0000 7f7bf17fa640 1 -- 172.21.15.114:0/2458761811 <== mon.1 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2858224613) 0x7f7bec052600 con 0x7f7bf4142690 2024-03-29T11:29:27.443 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.452+0000 7f7bf17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:27.444 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.452+0000 7f7bf17fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:29:27.444 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T11:29:27.444 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"b0de2b9b-c45a-44c7-b6ee-e83b39a1e108","created":"2024-03-29T11:20:49.420660+0000","modified":"2024-03-29T11:20:51.700167+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:51.700167+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a64785d3-6b11-499a-b706-cc827ab96a74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5882dadb-e7a8-4604-ba9a-418013c106c8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d7e8c409-bc66-4e4a-bbdb-420bad31388e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0682989-3595-4cd6-90a5-ab9111c278e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"67068120-e3ff-4817-bd06-44feb7479a35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09e8b-8a85-4997-bdea-75d754e6ee67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2f0d9ead-f47f-4cb3-92a4-0a5ad23cd831","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fdb6a25f-6bd8-4c65-9b9b-b460ba7673a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:29:27.447 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.456+0000 7f7bfb217640 10 librados: watch_flush enter 2024-03-29T11:29:27.447 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.456+0000 7f7bfb217640 10 librados: watch_flush exit 2024-03-29T11:29:27.447 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.456+0000 7f7bfb217640 20 client.4501.objecter shutdown clearing up homeless session... 2024-03-29T11:29:27.447 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.456+0000 7f7bfb217640 10 client.4501.objecter successfully canceled tick 2024-03-29T11:29:27.449 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.457+0000 7f7bfb217640 1 -- 172.21.15.114:0/2458761811 >> v1:172.21.15.114:6816/4038755055 conn(0x7f7be006dc90 legacy=0x7f7be0070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:27.449 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.457+0000 7f7bfb217640 10 monclient: shutdown 2024-03-29T11:29:27.449 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.458+0000 7f7bfb217640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:27.449 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.458+0000 7f7bfb217640 1 -- 172.21.15.114:0/2458761811 >> v1:172.21.15.176:6789/0 conn(0x7f7bf4142690 legacy=0x7f7bf414dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:27.450 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.459+0000 7f7bfb217640 1 -- 172.21.15.114:0/2458761811 shutdown_connections 2024-03-29T11:29:27.450 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.459+0000 7f7bfb217640 1 -- 172.21.15.114:0/2458761811 >> 172.21.15.114:0/2458761811 conn(0x7f7bf4066a00 msgr2=0x7f7bf4066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:27.450 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.459+0000 7f7bfb217640 1 -- 172.21.15.114:0/2458761811 shutdown_connections 2024-03-29T11:29:27.450 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.459+0000 7f7bfb217640 1 -- 172.21.15.114:0/2458761811 wait complete. 2024-03-29T11:29:27.450 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:27.459+0000 7f7bfb217640 1 librados: shutdown 2024-03-29T11:29:27.470 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:29:34.277 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:29:34.426 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.435+0000 7fe4be39d640 10 monclient: get_monmap_and_config 2024-03-29T11:29:34.427 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.435+0000 7fe4be39d640 10 monclient: build_initial_monmap 2024-03-29T11:29:34.427 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.435+0000 7fe4be39d640 10 monclient: monmap: 2024-03-29T11:29:34.427 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:29:34.427 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:29:34.427 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:29:34.436390+0000 2024-03-29T11:29:34.427 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:29:34.436390+0000 2024-03-29T11:29:34.427 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:34.427 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:29:34.427 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:29:34.427 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T11:29:34.427 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T11:29:34.427 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:29:34.428 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.436+0000 7fe4be39d640 1 Processor -- start 2024-03-29T11:29:34.428 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.436+0000 7fe4be39d640 1 -- start start 2024-03-29T11:29:34.428 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.436+0000 7fe4be39d640 10 monclient: init 2024-03-29T11:29:34.429 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.437+0000 7fe4be39d640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:34.429 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.437+0000 7fe4be39d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:29:34.429 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.437+0000 7fe4b7fff640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fe4b81422b0 0x7fe4b813f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:49784/0 (socket says 172.21.15.114:49784) 2024-03-29T11:29:34.429 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.437+0000 7fe4b7fff640 1 -- 172.21.15.114:0/2125185699 learned_addr learned my addr 172.21.15.114:0/2125185699 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:29:34.430 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.437+0000 7fe4be39d640 10 monclient(hunting): picked mon.noname-a con 0x7fe4b81422b0 addr v1:172.21.15.114:6789/0 2024-03-29T11:29:34.430 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.437+0000 7fe4be39d640 10 monclient(hunting): picked mon.noname-b con 0x7fe4b8142690 addr v1:172.21.15.114:6790/0 2024-03-29T11:29:34.430 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.437+0000 7fe4be39d640 10 monclient(hunting): picked mon.noname-c con 0x7fe4b80b17d0 addr v1:172.21.15.176:6789/0 2024-03-29T11:29:34.430 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.437+0000 7fe4be39d640 1 -- 172.21.15.114:0/2125185699 --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4b8142c80 con 0x7fe4b81422b0 2024-03-29T11:29:34.430 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.437+0000 7fe4be39d640 1 -- 172.21.15.114:0/2125185699 --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4b80aae40 con 0x7fe4b80b17d0 2024-03-29T11:29:34.430 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.437+0000 7fe4be39d640 1 -- 172.21.15.114:0/2125185699 --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4b813fbd0 con 0x7fe4b8142690 2024-03-29T11:29:34.430 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.437+0000 7fe4be39d640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:34.430 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.437+0000 7fe4be39d640 10 monclient(hunting): authenticate will time out at 1444.620014s 2024-03-29T11:29:34.430 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.437+0000 7fe4b6ffd640 1 -- 172.21.15.114:0/2125185699 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1228361673 0 0) 0x7fe4b8142c80 con 0x7fe4b81422b0 2024-03-29T11:29:34.430 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.437+0000 7fe4b6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:29:34.430 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.437+0000 7fe4b6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:34.430 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.437+0000 7fe4b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:34.430 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.437+0000 7fe4b6ffd640 10 monclient(hunting): my global_id is 4530 2024-03-29T11:29:34.431 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.437+0000 7fe4b6ffd640 1 -- 172.21.15.114:0/2125185699 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4a0003520 con 0x7fe4b81422b0 2024-03-29T11:29:34.432 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.438+0000 7fe4b6ffd640 1 -- 172.21.15.114:0/2125185699 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 773496286 0 0) 0x7fe4b80aae40 con 0x7fe4b80b17d0 2024-03-29T11:29:34.432 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.438+0000 7fe4b6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:29:34.432 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.438+0000 7fe4b6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:34.432 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.438+0000 7fe4b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:34.432 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.438+0000 7fe4b6ffd640 10 monclient(hunting): my global_id is 4504 2024-03-29T11:29:34.432 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.438+0000 7fe4b6ffd640 1 -- 172.21.15.114:0/2125185699 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4b8142c80 con 0x7fe4b80b17d0 2024-03-29T11:29:34.432 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.438+0000 7fe4b6ffd640 1 -- 172.21.15.114:0/2125185699 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4040285907 0 0) 0x7fe4b813fbd0 con 0x7fe4b8142690 2024-03-29T11:29:34.432 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.438+0000 7fe4b6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:29:34.432 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.438+0000 7fe4b6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:34.432 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.438+0000 7fe4b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:34.433 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.438+0000 7fe4b6ffd640 10 monclient(hunting): my global_id is 4565 2024-03-29T11:29:34.433 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.438+0000 7fe4b6ffd640 1 -- 172.21.15.114:0/2125185699 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4b80aae40 con 0x7fe4b8142690 2024-03-29T11:29:34.433 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.438+0000 7fe4b6ffd640 1 -- 172.21.15.114:0/2125185699 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 287818743 0 0) 0x7fe4a0003520 con 0x7fe4b81422b0 2024-03-29T11:29:34.433 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.438+0000 7fe4b6ffd640 1 -- 172.21.15.114:0/2125185699 >> v1:172.21.15.114:6790/0 conn(0x7fe4b8142690 legacy=0x7fe4b80b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:34.433 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.438+0000 7fe4b6ffd640 1 -- 172.21.15.114:0/2125185699 >> v1:172.21.15.176:6789/0 conn(0x7fe4b80b17d0 legacy=0x7fe4b80b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:34.433 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.438+0000 7fe4b6ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:29:34.433 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.438+0000 7fe4b6ffd640 1 monclient: found mon.noname-a 2024-03-29T11:29:34.433 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.438+0000 7fe4b6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:34.433 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.438+0000 7fe4b6ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.114:6789/0 2024-03-29T11:29:34.433 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.438+0000 7fe4b6ffd640 1 -- 172.21.15.114:0/2125185699 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4b8150930 con 0x7fe4b81422b0 2024-03-29T11:29:34.433 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.438+0000 7fe4b6ffd640 10 monclient: _finish_auth 0 2024-03-29T11:29:34.433 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.438+0000 7fe4b6ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:29:34.433 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.438+0000 7fe4b6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:34.433 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.438+0000 7fe4b6ffd640 1 -- 172.21.15.114:0/2125185699 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fe4b0001990 con 0x7fe4b81422b0 2024-03-29T11:29:34.433 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.438+0000 7fe4b6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:34.434 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.438+0000 7fe4b6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:29:34.434 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.438+0000 7fe4b6ffd640 10 monclient: dump: 2024-03-29T11:29:34.434 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:29:34.434 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:29:34.434 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:29:34.434 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:29:34.434 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:34.434 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:29:34.434 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:29:34.434 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:29:34.434 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:29:34.434 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:29:34.434 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.439+0000 7fe4be39d640 5 monclient: authenticate success, global_id 4530 2024-03-29T11:29:34.435 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.439+0000 7fe4be39d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:34.435 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.439+0000 7fe4b6ffd640 1 -- 172.21.15.114:0/2125185699 <== mon.0 v1:172.21.15.114:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe4b0001af0 con 0x7fe4b81422b0 2024-03-29T11:29:34.435 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.439+0000 7fe4b6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:34.435 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.439+0000 7fe4b6ffd640 1 -- 172.21.15.114:0/2125185699 <== mon.0 v1:172.21.15.114:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fe4b0001f40 con 0x7fe4b81422b0 2024-03-29T11:29:34.435 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.439+0000 7fe4b6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:34.435 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.439+0000 7fe4b6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:29:34.435 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.439+0000 7fe4b6ffd640 10 monclient: dump: 2024-03-29T11:29:34.435 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:29:34.435 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:29:34.435 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:29:34.435 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:29:34.435 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:34.435 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:29:34.435 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:29:34.435 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:29:34.435 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:29:34.436 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:29:34.436 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.439+0000 7fe4be39d640 10 monclient: get_monmap_and_config success 2024-03-29T11:29:34.436 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.439+0000 7fe4be39d640 4 set_mon_vals no callback set 2024-03-29T11:29:34.436 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.441+0000 7fe4be39d640 10 monclient: shutdown 2024-03-29T11:29:34.436 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.441+0000 7fe4be39d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:34.436 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.441+0000 7fe4be39d640 1 -- 172.21.15.114:0/2125185699 >> v1:172.21.15.114:6789/0 conn(0x7fe4b81422b0 legacy=0x7fe4b813f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:34.437 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.441+0000 7fe4be39d640 1 -- 172.21.15.114:0/2125185699 shutdown_connections 2024-03-29T11:29:34.437 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.441+0000 7fe4be39d640 1 -- 172.21.15.114:0/2125185699 >> 172.21.15.114:0/2125185699 conn(0x7fe4b8066a00 msgr2=0x7fe4b8066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:34.437 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.441+0000 7fe4be39d640 1 -- 172.21.15.114:0/2125185699 shutdown_connections 2024-03-29T11:29:34.437 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.441+0000 7fe4be39d640 1 -- 172.21.15.114:0/2125185699 wait complete. 2024-03-29T11:29:34.438 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.442+0000 7fe4be39d640 10 monclient: build_initial_monmap 2024-03-29T11:29:34.438 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.443+0000 7fe4be39d640 10 monclient: monmap: 2024-03-29T11:29:34.438 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:29:34.438 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:29:34.438 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T11:29:34.438 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T11:29:34.438 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:34.438 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:29:34.438 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:29:34.438 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T11:29:34.438 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T11:29:34.438 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:29:34.438 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.444+0000 7fe4be39d640 1 librados: starting msgr at 2024-03-29T11:29:34.438 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.444+0000 7fe4be39d640 1 librados: starting objecter 2024-03-29T11:29:34.438 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.444+0000 7fe4be39d640 1 Processor -- start 2024-03-29T11:29:34.439 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.445+0000 7fe4be39d640 1 -- start start 2024-03-29T11:29:34.439 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.445+0000 7fe4be39d640 1 librados: setting wanted keys 2024-03-29T11:29:34.439 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.445+0000 7fe4be39d640 1 librados: calling monclient init 2024-03-29T11:29:34.439 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.445+0000 7fe4be39d640 10 monclient: init 2024-03-29T11:29:34.448 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.447+0000 7fe4be39d640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:34.448 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.447+0000 7fe4be39d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:29:34.448 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.447+0000 7fe4be39d640 10 monclient(hunting): picked mon.noname-c con 0x7fe4b80b17d0 addr v1:172.21.15.114:6790/0 2024-03-29T11:29:34.448 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.447+0000 7fe4be39d640 10 monclient(hunting): picked mon.noname-b con 0x7fe4b81422b0 addr v1:172.21.15.176:6789/0 2024-03-29T11:29:34.448 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.448+0000 7fe4be39d640 10 monclient(hunting): picked mon.noname-a con 0x7fe4b8142690 addr v1:172.21.15.114:6789/0 2024-03-29T11:29:34.448 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.448+0000 7fe4be39d640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4b80acd30 con 0x7fe4b8142690 2024-03-29T11:29:34.449 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.448+0000 7fe4be39d640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4b80adab0 con 0x7fe4b81422b0 2024-03-29T11:29:34.449 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.448+0000 7fe4be39d640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4b80add20 con 0x7fe4b80b17d0 2024-03-29T11:29:34.449 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.448+0000 7fe4be39d640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:34.449 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.448+0000 7fe4be39d640 10 monclient(hunting): authenticate will time out at 1444.630261s 2024-03-29T11:29:34.449 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.448+0000 7fe4b77fe640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7fe4b81422b0 0x7fe4b812f050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.114:41358/0 (socket says 172.21.15.114:41358) 2024-03-29T11:29:34.449 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.448+0000 7fe4b77fe640 1 -- 172.21.15.114:0/1085581683 learned_addr learned my addr 172.21.15.114:0/1085581683 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:29:34.449 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.448+0000 7fe4b4ff9640 10 client.?.objecter ms_handle_connect 0x7fe4b81422b0 2024-03-29T11:29:34.449 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.448+0000 7fe4b4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:34.449 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.448+0000 7fe4b4ff9640 10 client.?.objecter ms_handle_connect 0x7fe4b8142690 2024-03-29T11:29:34.449 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.448+0000 7fe4b4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:34.449 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.448+0000 7fe4b4ff9640 10 client.?.objecter ms_handle_connect 0x7fe4b80b17d0 2024-03-29T11:29:34.449 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.448+0000 7fe4b4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:34.449 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.448+0000 7fe4b4ff9640 1 -- 172.21.15.114:0/1085581683 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2267720244 0 0) 0x7fe4b80acd30 con 0x7fe4b8142690 2024-03-29T11:29:34.449 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.448+0000 7fe4b4ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:29:34.449 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.448+0000 7fe4b4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:34.450 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.449+0000 7fe4b4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:34.450 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.449+0000 7fe4b4ff9640 10 monclient(hunting): my global_id is 4533 2024-03-29T11:29:34.450 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.449+0000 7fe4b4ff9640 1 -- 172.21.15.114:0/1085581683 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4a4004eb0 con 0x7fe4b8142690 2024-03-29T11:29:34.450 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.449+0000 7fe4b4ff9640 1 -- 172.21.15.114:0/1085581683 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1750841419 0 0) 0x7fe4b80add20 con 0x7fe4b80b17d0 2024-03-29T11:29:34.450 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.449+0000 7fe4b4ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:29:34.450 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.449+0000 7fe4b4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:34.450 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.449+0000 7fe4b4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:34.450 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.449+0000 7fe4b4ff9640 10 monclient(hunting): my global_id is 4568 2024-03-29T11:29:34.450 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.449+0000 7fe4b4ff9640 1 -- 172.21.15.114:0/1085581683 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4b80acd30 con 0x7fe4b80b17d0 2024-03-29T11:29:34.450 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.449+0000 7fe4b4ff9640 1 -- 172.21.15.114:0/1085581683 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1493968173 0 0) 0x7fe4b80adab0 con 0x7fe4b81422b0 2024-03-29T11:29:34.450 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.449+0000 7fe4b4ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:29:34.450 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.449+0000 7fe4b4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:34.450 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.449+0000 7fe4b4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:34.450 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.449+0000 7fe4b4ff9640 10 monclient(hunting): my global_id is 4507 2024-03-29T11:29:34.450 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.449+0000 7fe4b4ff9640 1 -- 172.21.15.114:0/1085581683 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4b80add20 con 0x7fe4b81422b0 2024-03-29T11:29:34.451 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.449+0000 7fe4b4ff9640 1 -- 172.21.15.114:0/1085581683 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1662651595 0 0) 0x7fe4b80acd30 con 0x7fe4b80b17d0 2024-03-29T11:29:34.451 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.449+0000 7fe4b4ff9640 1 -- 172.21.15.114:0/1085581683 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe4b80adab0 con 0x7fe4b80b17d0 2024-03-29T11:29:34.451 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.449+0000 7fe4b4ff9640 1 -- 172.21.15.114:0/1085581683 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fe4b0001e00 con 0x7fe4b80b17d0 2024-03-29T11:29:34.451 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.449+0000 7fe4b4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:34.451 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.449+0000 7fe4b4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:29:34.451 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.449+0000 7fe4b4ff9640 10 monclient(hunting): dump: 2024-03-29T11:29:34.451 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:29:34.451 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:29:34.451 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:29:34.451 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:29:34.451 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:34.451 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:29:34.451 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:29:34.451 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:29:34.452 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:29:34.452 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:29:34.452 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.449+0000 7fe4b4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:29:34.452 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.449+0000 7fe4b4ff9640 1 -- 172.21.15.114:0/1085581683 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1688264846 0 0) 0x7fe4a4004eb0 con 0x7fe4b8142690 2024-03-29T11:29:34.452 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.449+0000 7fe4b4ff9640 1 -- 172.21.15.114:0/1085581683 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe4b80acd30 con 0x7fe4b8142690 2024-03-29T11:29:34.452 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.449+0000 7fe4b4ff9640 1 -- 172.21.15.114:0/1085581683 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fe4a80018b0 con 0x7fe4b8142690 2024-03-29T11:29:34.452 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.449+0000 7fe4b4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:34.452 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.449+0000 7fe4b4ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:29:34.452 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.449+0000 7fe4b4ff9640 10 monclient(hunting): dump: 2024-03-29T11:29:34.452 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:29:34.452 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:29:34.452 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:29:34.452 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:29:34.452 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:34.452 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:29:34.453 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:29:34.453 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:29:34.453 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:29:34.453 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:29:34.453 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.449+0000 7fe4b4ff9640 1 -- 172.21.15.114:0/1085581683 <== mon.2 v1:172.21.15.114:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 579058033 0 0) 0x7fe4b80adab0 con 0x7fe4b80b17d0 2024-03-29T11:29:34.453 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.449+0000 7fe4b4ff9640 1 -- 172.21.15.114:0/1085581683 >> v1:172.21.15.176:6789/0 conn(0x7fe4b81422b0 legacy=0x7fe4b812f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:34.453 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.449+0000 7fe4b4ff9640 1 -- 172.21.15.114:0/1085581683 >> v1:172.21.15.114:6789/0 conn(0x7fe4b8142690 legacy=0x7fe4b8131c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:34.453 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.450+0000 7fe4b4ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:29:34.453 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.450+0000 7fe4b4ff9640 1 monclient: found mon.c 2024-03-29T11:29:34.453 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.450+0000 7fe4b4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:34.453 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.450+0000 7fe4b4ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:29:34.453 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.450+0000 7fe4b4ff9640 1 -- 172.21.15.114:0/1085581683 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4b80ab9a0 con 0x7fe4b80b17d0 2024-03-29T11:29:34.453 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.450+0000 7fe4b4ff9640 10 monclient: _finish_auth 0 2024-03-29T11:29:34.453 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.450+0000 7fe4b4ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:29:34.453 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.450+0000 7fe4b4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:34.454 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.450+0000 7fe4be39d640 5 monclient: authenticate success, global_id 4568 2024-03-29T11:29:34.454 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.450+0000 7fe4be39d640 10 monclient: _renew_subs 2024-03-29T11:29:34.454 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.450+0000 7fe4be39d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:29:34.454 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.450+0000 7fe4be39d640 1 -- 172.21.15.114:0/1085581683 --> v1:172.21.15.114:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4b80abbf0 con 0x7fe4b80b17d0 2024-03-29T11:29:34.454 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.450+0000 7fe4be39d640 10 client.4568.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:29:34.454 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.450+0000 7fe4be39d640 10 monclient: _renew_subs 2024-03-29T11:29:34.454 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.450+0000 7fe4be39d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:29:34.454 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.450+0000 7fe4be39d640 1 -- 172.21.15.114:0/1085581683 --> v1:172.21.15.114:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4b812f940 con 0x7fe4b80b17d0 2024-03-29T11:29:34.454 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.450+0000 7fe4be39d640 1 librados: init done 2024-03-29T11:29:34.454 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.451+0000 7fe4b4ff9640 1 -- 172.21.15.114:0/1085581683 <== mon.2 v1:172.21.15.114:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe4b0001f60 con 0x7fe4b80b17d0 2024-03-29T11:29:34.454 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.451+0000 7fe4b4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:34.454 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.451+0000 7fe4b4ff9640 1 -- 172.21.15.114:0/1085581683 <== mon.2 v1:172.21.15.114:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fe4b0004e10 con 0x7fe4b80b17d0 2024-03-29T11:29:34.454 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.451+0000 7fe4b4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:34.454 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.451+0000 7fe4b4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:29:34.455 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.451+0000 7fe4b4ff9640 10 monclient: dump: 2024-03-29T11:29:34.455 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:29:34.455 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:29:34.455 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:29:34.455 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:29:34.455 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:34.455 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:29:34.455 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:29:34.455 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:29:34.455 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:29:34.455 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:29:34.455 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.452+0000 7fe4be39d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:34.455 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.452+0000 7fe4b67fc640 4 set_mon_vals no callback set 2024-03-29T11:29:34.455 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.453+0000 7fe4b4ff9640 1 -- 172.21.15.114:0/1085581683 <== mon.2 v1:172.21.15.114:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1594292461 0 0) 0x7fe4b001ade0 con 0x7fe4b80b17d0 2024-03-29T11:29:34.455 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.453+0000 7fe4be39d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:34.456 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.453+0000 7fe4be39d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:29:34.456 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.453+0000 7fe4be39d640 1 -- 172.21.15.114:0/1085581683 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4b80acec0 con 0x7fe4b80b17d0 2024-03-29T11:29:34.456 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.454+0000 7fe4b77fe640 10 monclient: get_auth_request con 0x7fe4a406df70 auth_method 0 2024-03-29T11:29:34.456 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.454+0000 7fe4b4ff9640 1 -- 172.21.15.114:0/1085581683 <== mon.2 v1:172.21.15.114:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3238987313 0 0) 0x7fe4b0084d50 con 0x7fe4b80b17d0 2024-03-29T11:29:34.456 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.454+0000 7fe4b4ff9640 10 client.4568.objecter ms_dispatch 0x7fe4b8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:29:34.456 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.454+0000 7fe4b4ff9640 3 client.4568.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:29:34.456 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.454+0000 7fe4b4ff9640 3 client.4568.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:29:34.456 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.454+0000 7fe4b4ff9640 20 client.4568.objecter dump_active .. 0 homeless 2024-03-29T11:29:34.456 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.454+0000 7fe4b4ff9640 10 client.4568.objecter ms_handle_connect 0x7fe4a406df70 2024-03-29T11:29:34.456 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.464+0000 7fe4b4ff9640 1 -- 172.21.15.114:0/1085581683 <== mon.2 v1:172.21.15.114:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe4b0050630 con 0x7fe4b80b17d0 2024-03-29T11:29:34.456 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.464+0000 7fe4b4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:34.457 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.464+0000 7fe4b4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:29:34.820 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.828+0000 7fe4be39d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:34.820 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.828+0000 7fe4be39d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:34.820 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.828+0000 7fe4be39d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:29:34.820 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.828+0000 7fe4be39d640 1 -- 172.21.15.114:0/1085581683 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe4b8075f10 con 0x7fe4b80b17d0 2024-03-29T11:29:34.821 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.830+0000 7fe4b4ff9640 1 -- 172.21.15.114:0/1085581683 <== mon.2 v1:172.21.15.114:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2858224613) 0x7fe4b0052560 con 0x7fe4b80b17d0 2024-03-29T11:29:34.822 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.830+0000 7fe4b4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:34.822 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.830+0000 7fe4b4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:29:34.822 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T11:29:34.822 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"b0de2b9b-c45a-44c7-b6ee-e83b39a1e108","created":"2024-03-29T11:20:49.420660+0000","modified":"2024-03-29T11:20:51.700167+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:51.700167+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a64785d3-6b11-499a-b706-cc827ab96a74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5882dadb-e7a8-4604-ba9a-418013c106c8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d7e8c409-bc66-4e4a-bbdb-420bad31388e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0682989-3595-4cd6-90a5-ab9111c278e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"67068120-e3ff-4817-bd06-44feb7479a35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09e8b-8a85-4997-bdea-75d754e6ee67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2f0d9ead-f47f-4cb3-92a4-0a5ad23cd831","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fdb6a25f-6bd8-4c65-9b9b-b460ba7673a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:29:34.825 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.833+0000 7fe4be39d640 10 librados: watch_flush enter 2024-03-29T11:29:34.825 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.833+0000 7fe4be39d640 10 librados: watch_flush exit 2024-03-29T11:29:34.825 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.833+0000 7fe4be39d640 20 client.4568.objecter shutdown clearing up homeless session... 2024-03-29T11:29:34.825 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.833+0000 7fe4be39d640 10 client.4568.objecter successfully canceled tick 2024-03-29T11:29:34.826 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.834+0000 7fe4be39d640 1 -- 172.21.15.114:0/1085581683 >> v1:172.21.15.114:6816/4038755055 conn(0x7fe4a406df70 legacy=0x7fe4a4070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:34.826 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.834+0000 7fe4be39d640 10 monclient: shutdown 2024-03-29T11:29:34.826 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.834+0000 7fe4be39d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:34.826 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.834+0000 7fe4be39d640 1 -- 172.21.15.114:0/1085581683 >> v1:172.21.15.114:6790/0 conn(0x7fe4b80b17d0 legacy=0x7fe4b80ac620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:34.826 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.834+0000 7fe4be39d640 1 -- 172.21.15.114:0/1085581683 shutdown_connections 2024-03-29T11:29:34.826 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.834+0000 7fe4be39d640 1 -- 172.21.15.114:0/1085581683 >> 172.21.15.114:0/1085581683 conn(0x7fe4b8066a00 msgr2=0x7fe4b813f290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:34.826 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.834+0000 7fe4be39d640 1 -- 172.21.15.114:0/1085581683 shutdown_connections 2024-03-29T11:29:34.826 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.834+0000 7fe4be39d640 1 -- 172.21.15.114:0/1085581683 wait complete. 2024-03-29T11:29:34.826 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:34.834+0000 7fe4be39d640 1 librados: shutdown 2024-03-29T11:29:34.840 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:29:41.648 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:29:41.799 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.807+0000 7f2ccf29a640 10 monclient: get_monmap_and_config 2024-03-29T11:29:41.799 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.807+0000 7f2ccf29a640 10 monclient: build_initial_monmap 2024-03-29T11:29:41.799 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.807+0000 7f2ccf29a640 10 monclient: monmap: 2024-03-29T11:29:41.799 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:29:41.799 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:29:41.800 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:29:41.808082+0000 2024-03-29T11:29:41.800 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:29:41.808082+0000 2024-03-29T11:29:41.800 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:41.800 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:29:41.800 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:29:41.800 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T11:29:41.800 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T11:29:41.800 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:29:41.804 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.810+0000 7f2ccf29a640 1 Processor -- start 2024-03-29T11:29:41.804 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.810+0000 7f2ccf29a640 1 -- start start 2024-03-29T11:29:41.804 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.810+0000 7f2ccf29a640 10 monclient: init 2024-03-29T11:29:41.804 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.811+0000 7f2ccf29a640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:41.804 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.811+0000 7f2ccf29a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:29:41.804 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.811+0000 7f2ccf29a640 10 monclient(hunting): picked mon.noname-a con 0x7f2cc80a9930 addr v1:172.21.15.114:6789/0 2024-03-29T11:29:41.804 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.811+0000 7f2ccf29a640 10 monclient(hunting): picked mon.noname-c con 0x7f2cc8149dc0 addr v1:172.21.15.176:6789/0 2024-03-29T11:29:41.804 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.811+0000 7f2ccf29a640 10 monclient(hunting): picked mon.noname-b con 0x7f2cc814ff40 addr v1:172.21.15.114:6790/0 2024-03-29T11:29:41.804 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.811+0000 7f2ccf29a640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cc80aae60 con 0x7f2cc80a9930 2024-03-29T11:29:41.805 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.811+0000 7f2ccf29a640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cc80b4310 con 0x7f2cc8149dc0 2024-03-29T11:29:41.805 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.811+0000 7f2ccf29a640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cc80afdd0 con 0x7f2cc814ff40 2024-03-29T11:29:41.805 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.811+0000 7f2ccf29a640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:41.805 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.811+0000 7f2ccf29a640 10 monclient(hunting): authenticate will time out at 1451.993533s 2024-03-29T11:29:41.806 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.814+0000 7f2ccd837640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f2cc814ff40 0x7f2cc8152330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:37019/0 (socket says 172.21.15.114:37019) 2024-03-29T11:29:41.806 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.814+0000 7f2ccd837640 1 -- 172.21.15.114:0/4121774233 learned_addr learned my addr 172.21.15.114:0/4121774233 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:29:41.809 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.816+0000 7f2cbffff640 1 -- 172.21.15.114:0/4121774233 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1732267055 0 0) 0x7f2cc80afdd0 con 0x7f2cc814ff40 2024-03-29T11:29:41.809 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.816+0000 7f2cbffff640 10 monclient(hunting): _negotiate 2024-03-29T11:29:41.809 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.816+0000 7f2cbffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:41.810 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.816+0000 7f2cbffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:41.810 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.816+0000 7f2cbffff640 10 monclient(hunting): my global_id is 4571 2024-03-29T11:29:41.810 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.816+0000 7f2cbffff640 1 -- 172.21.15.114:0/4121774233 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2cb4003520 con 0x7f2cc814ff40 2024-03-29T11:29:41.810 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.816+0000 7f2cbffff640 1 -- 172.21.15.114:0/4121774233 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3836674320 0 0) 0x7f2cc80aae60 con 0x7f2cc80a9930 2024-03-29T11:29:41.810 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.816+0000 7f2cbffff640 10 monclient(hunting): _negotiate 2024-03-29T11:29:41.810 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.816+0000 7f2cbffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:41.810 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.816+0000 7f2cbffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:41.810 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.817+0000 7f2cbffff640 10 monclient(hunting): my global_id is 4536 2024-03-29T11:29:41.810 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.817+0000 7f2cbffff640 1 -- 172.21.15.114:0/4121774233 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2cc80afdd0 con 0x7f2cc80a9930 2024-03-29T11:29:41.810 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.817+0000 7f2cbffff640 1 -- 172.21.15.114:0/4121774233 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2520962070 0 0) 0x7f2cb4003520 con 0x7f2cc814ff40 2024-03-29T11:29:41.810 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.817+0000 7f2cbffff640 1 -- 172.21.15.114:0/4121774233 >> v1:172.21.15.176:6789/0 conn(0x7f2cc8149dc0 legacy=0x7f2cc814a240 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:29:41.813 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.818+0000 7f2cbffff640 1 -- 172.21.15.114:0/4121774233 >> v1:172.21.15.114:6789/0 conn(0x7f2cc80a9930 legacy=0x7f2cc8143a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:41.813 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.818+0000 7f2cbffff640 10 monclient: _finish_hunting 0 2024-03-29T11:29:41.813 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.818+0000 7f2cbffff640 1 monclient: found mon.noname-b 2024-03-29T11:29:41.813 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.818+0000 7f2cbffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:41.813 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.818+0000 7f2cbffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.114:6790/0 2024-03-29T11:29:41.813 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.818+0000 7f2cbffff640 1 -- 172.21.15.114:0/4121774233 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cc814a950 con 0x7f2cc814ff40 2024-03-29T11:29:41.814 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.818+0000 7f2cbffff640 10 monclient: _finish_auth 0 2024-03-29T11:29:41.814 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.818+0000 7f2cbffff640 10 monclient: _check_auth_tickets 2024-03-29T11:29:41.814 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.818+0000 7f2cbffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:41.814 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.818+0000 7f2ccf29a640 5 monclient: authenticate success, global_id 4571 2024-03-29T11:29:41.814 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.818+0000 7f2ccf29a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:41.814 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.818+0000 7f2cbffff640 1 -- 172.21.15.114:0/4121774233 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f2cc4001690 con 0x7f2cc814ff40 2024-03-29T11:29:41.814 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.818+0000 7f2cbffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:41.814 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.818+0000 7f2cbffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:29:41.814 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.818+0000 7f2cbffff640 10 monclient: dump: 2024-03-29T11:29:41.814 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:29:41.814 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:29:41.814 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:29:41.814 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:29:41.815 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:41.815 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:29:41.815 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:29:41.815 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:29:41.815 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:29:41.815 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:29:41.815 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.818+0000 7f2ccf29a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:41.815 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.818+0000 7f2cbffff640 1 -- 172.21.15.114:0/4121774233 <== mon.2 v1:172.21.15.114:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2cc40018b0 con 0x7f2cc814ff40 2024-03-29T11:29:41.815 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.818+0000 7f2cbffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:41.815 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.818+0000 7f2ccf29a640 10 monclient: get_monmap_and_config success 2024-03-29T11:29:41.815 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.818+0000 7f2ccf29a640 4 set_mon_vals no callback set 2024-03-29T11:29:41.815 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.819+0000 7f2ccf29a640 10 monclient: shutdown 2024-03-29T11:29:41.815 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.819+0000 7f2ccf29a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:41.815 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.819+0000 7f2ccf29a640 1 -- 172.21.15.114:0/4121774233 >> v1:172.21.15.114:6790/0 conn(0x7f2cc814ff40 legacy=0x7f2cc8152330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:41.817 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.822+0000 7f2ccf29a640 1 -- 172.21.15.114:0/4121774233 shutdown_connections 2024-03-29T11:29:41.817 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.822+0000 7f2ccf29a640 1 -- 172.21.15.114:0/4121774233 >> 172.21.15.114:0/4121774233 conn(0x7f2cc80b35e0 msgr2=0x7f2cc80b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:41.817 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.823+0000 7f2ccf29a640 1 -- 172.21.15.114:0/4121774233 shutdown_connections 2024-03-29T11:29:41.817 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.823+0000 7f2ccf29a640 1 -- 172.21.15.114:0/4121774233 wait complete. 2024-03-29T11:29:41.818 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.824+0000 7f2ccf29a640 10 monclient: build_initial_monmap 2024-03-29T11:29:41.818 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.824+0000 7f2ccf29a640 10 monclient: monmap: 2024-03-29T11:29:41.818 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:29:41.818 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:29:41.818 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T11:29:41.818 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T11:29:41.818 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:41.818 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:29:41.818 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:29:41.818 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T11:29:41.818 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T11:29:41.818 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:29:41.818 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.824+0000 7f2ccf29a640 1 librados: starting msgr at 2024-03-29T11:29:41.818 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.824+0000 7f2ccf29a640 1 librados: starting objecter 2024-03-29T11:29:41.819 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.824+0000 7f2ccf29a640 1 Processor -- start 2024-03-29T11:29:41.819 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.824+0000 7f2ccf29a640 1 -- start start 2024-03-29T11:29:41.821 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.824+0000 7f2ccf29a640 1 librados: setting wanted keys 2024-03-29T11:29:41.821 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.824+0000 7f2ccf29a640 1 librados: calling monclient init 2024-03-29T11:29:41.821 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.824+0000 7f2ccf29a640 10 monclient: init 2024-03-29T11:29:41.821 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.825+0000 7f2ccf29a640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:41.821 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.825+0000 7f2ccf29a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:29:41.821 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.825+0000 7f2ccf29a640 10 monclient(hunting): picked mon.noname-b con 0x7f2cc80a9930 addr v1:172.21.15.176:6789/0 2024-03-29T11:29:41.821 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.825+0000 7f2ccf29a640 10 monclient(hunting): picked mon.noname-a con 0x7f2cc8149dc0 addr v1:172.21.15.114:6789/0 2024-03-29T11:29:41.821 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.825+0000 7f2ccf29a640 10 monclient(hunting): picked mon.noname-c con 0x7f2cc814ff40 addr v1:172.21.15.114:6790/0 2024-03-29T11:29:41.821 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.825+0000 7f2ccf29a640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cc813bd30 con 0x7f2cc8149dc0 2024-03-29T11:29:41.821 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.825+0000 7f2ccf29a640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cc8145870 con 0x7f2cc80a9930 2024-03-29T11:29:41.821 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.825+0000 7f2ccf29a640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cc8148bc0 con 0x7f2cc814ff40 2024-03-29T11:29:41.821 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.825+0000 7f2ccf29a640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:41.821 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.825+0000 7f2ccf29a640 10 monclient(hunting): authenticate will time out at 1452.007526s 2024-03-29T11:29:41.824 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.830+0000 7f2ccc835640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7f2cc8149dc0 0x7f2cc813af10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:49027/0 (socket says 172.21.15.114:49027) 2024-03-29T11:29:41.824 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.830+0000 7f2ccc835640 1 -- 172.21.15.114:0/15185132 learned_addr learned my addr 172.21.15.114:0/15185132 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:29:41.824 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.832+0000 7f2cbdffb640 10 client.?.objecter ms_handle_connect 0x7f2cc8149dc0 2024-03-29T11:29:41.824 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.832+0000 7f2cbdffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:41.824 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.832+0000 7f2cbdffb640 10 client.?.objecter ms_handle_connect 0x7f2cc80a9930 2024-03-29T11:29:41.824 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.832+0000 7f2cbdffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:41.824 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.832+0000 7f2cbdffb640 1 -- 172.21.15.114:0/15185132 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1795437331 0 0) 0x7f2cc8145870 con 0x7f2cc80a9930 2024-03-29T11:29:41.825 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.832+0000 7f2cbdffb640 10 monclient(hunting): _negotiate 2024-03-29T11:29:41.825 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.832+0000 7f2cbdffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:41.825 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.832+0000 7f2cbdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:41.825 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.832+0000 7f2cbdffb640 10 monclient(hunting): my global_id is 4510 2024-03-29T11:29:41.826 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.832+0000 7f2cbdffb640 1 -- 172.21.15.114:0/15185132 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2cac004eb0 con 0x7f2cc80a9930 2024-03-29T11:29:41.832 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.833+0000 7f2cbdffb640 1 -- 172.21.15.114:0/15185132 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1149400625 0 0) 0x7f2cc813bd30 con 0x7f2cc8149dc0 2024-03-29T11:29:41.832 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.833+0000 7f2cbdffb640 10 monclient(hunting): _negotiate 2024-03-29T11:29:41.832 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.833+0000 7f2cbdffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:41.832 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.833+0000 7f2cbdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:41.832 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.833+0000 7f2cbdffb640 10 monclient(hunting): my global_id is 4539 2024-03-29T11:29:41.832 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.833+0000 7f2cbdffb640 1 -- 172.21.15.114:0/15185132 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2cc8145870 con 0x7f2cc8149dc0 2024-03-29T11:29:41.833 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.833+0000 7f2cbdffb640 10 client.?.objecter ms_handle_connect 0x7f2cc814ff40 2024-03-29T11:29:41.833 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.833+0000 7f2cbdffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:41.833 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.834+0000 7f2cbdffb640 1 -- 172.21.15.114:0/15185132 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3265081154 0 0) 0x7f2cac004eb0 con 0x7f2cc80a9930 2024-03-29T11:29:41.833 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.834+0000 7f2cbdffb640 1 -- 172.21.15.114:0/15185132 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2cac005490 con 0x7f2cc80a9930 2024-03-29T11:29:41.833 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.834+0000 7f2cbdffb640 1 -- 172.21.15.114:0/15185132 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f2cc0001c80 con 0x7f2cc80a9930 2024-03-29T11:29:41.833 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.834+0000 7f2cbdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:41.833 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.834+0000 7f2cbdffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:29:41.833 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.834+0000 7f2cbdffb640 10 monclient(hunting): dump: 2024-03-29T11:29:41.833 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:29:41.833 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:29:41.833 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:29:41.833 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:29:41.833 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:41.833 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:29:41.833 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:29:41.834 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:29:41.834 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:29:41.834 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:29:41.834 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.834+0000 7f2cbdffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:29:41.834 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.834+0000 7f2cbdffb640 1 -- 172.21.15.114:0/15185132 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4196449260 0 0) 0x7f2cc8145870 con 0x7f2cc8149dc0 2024-03-29T11:29:41.834 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.834+0000 7f2cbdffb640 1 -- 172.21.15.114:0/15185132 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2cac004eb0 con 0x7f2cc8149dc0 2024-03-29T11:29:41.834 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.834+0000 7f2cbdffb640 1 -- 172.21.15.114:0/15185132 <== mon.1 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2633886088 0 0) 0x7f2cac005490 con 0x7f2cc80a9930 2024-03-29T11:29:41.834 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.834+0000 7f2cbdffb640 1 -- 172.21.15.114:0/15185132 >> v1:172.21.15.114:6790/0 conn(0x7f2cc814ff40 legacy=0x7f2cc813b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:41.834 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.834+0000 7f2cbdffb640 1 -- 172.21.15.114:0/15185132 >> v1:172.21.15.114:6789/0 conn(0x7f2cc8149dc0 legacy=0x7f2cc813af10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:41.834 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.834+0000 7f2cbdffb640 10 monclient: _finish_hunting 0 2024-03-29T11:29:41.834 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.834+0000 7f2cbdffb640 1 monclient: found mon.b 2024-03-29T11:29:41.834 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.834+0000 7f2cbdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:41.834 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.834+0000 7f2cbdffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:29:41.834 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.834+0000 7f2cbdffb640 1 -- 172.21.15.114:0/15185132 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cc81475e0 con 0x7f2cc80a9930 2024-03-29T11:29:41.835 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.834+0000 7f2cbdffb640 10 monclient: _finish_auth 0 2024-03-29T11:29:41.835 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.834+0000 7f2cbdffb640 10 monclient: _check_auth_tickets 2024-03-29T11:29:41.835 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.834+0000 7f2cbdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:41.835 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.835+0000 7f2ccf29a640 5 monclient: authenticate success, global_id 4510 2024-03-29T11:29:41.835 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.835+0000 7f2ccf29a640 10 monclient: _renew_subs 2024-03-29T11:29:41.835 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.835+0000 7f2ccf29a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:29:41.835 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.835+0000 7f2ccf29a640 1 -- 172.21.15.114:0/15185132 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cc81487e0 con 0x7f2cc80a9930 2024-03-29T11:29:41.835 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.835+0000 7f2cbdffb640 1 -- 172.21.15.114:0/15185132 <== mon.1 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2cc00020b0 con 0x7f2cc80a9930 2024-03-29T11:29:41.835 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.835+0000 7f2cbdffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:41.835 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.835+0000 7f2cbdffb640 1 -- 172.21.15.114:0/15185132 <== mon.1 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f2cc0003b10 con 0x7f2cc80a9930 2024-03-29T11:29:41.835 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.835+0000 7f2cbdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:41.835 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.835+0000 7f2cbdffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:29:41.835 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.835+0000 7f2cbdffb640 10 monclient: dump: 2024-03-29T11:29:41.835 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:29:41.835 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:29:41.836 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:29:41.836 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:29:41.836 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:41.836 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:29:41.836 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:29:41.836 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:29:41.836 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:29:41.836 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:29:41.836 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.835+0000 7f2ccf29a640 10 client.4510.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:29:41.836 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.835+0000 7f2ccf29a640 10 monclient: _renew_subs 2024-03-29T11:29:41.836 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.835+0000 7f2ccf29a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:29:41.836 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.835+0000 7f2ccf29a640 1 -- 172.21.15.114:0/15185132 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cc81496d0 con 0x7f2cc80a9930 2024-03-29T11:29:41.836 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.835+0000 7f2ccf29a640 1 librados: init done 2024-03-29T11:29:41.836 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.835+0000 7f2cbeffd640 4 set_mon_vals no callback set 2024-03-29T11:29:41.836 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.836+0000 7f2ccf29a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:41.837 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.836+0000 7f2ccf29a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:41.837 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.837+0000 7f2ccf29a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:29:41.837 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.837+0000 7f2ccf29a640 1 -- 172.21.15.114:0/15185132 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cc8148a20 con 0x7f2cc80a9930 2024-03-29T11:29:41.837 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.838+0000 7f2cbdffb640 1 -- 172.21.15.114:0/15185132 <== mon.1 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1594292461 0 0) 0x7f2cc0019ae0 con 0x7f2cc80a9930 2024-03-29T11:29:41.837 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.838+0000 7f2cbdffb640 1 -- 172.21.15.114:0/15185132 <== mon.1 v1:172.21.15.176:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3238987313 0 0) 0x7f2cc0083ac0 con 0x7f2cc80a9930 2024-03-29T11:29:41.837 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.838+0000 7f2cbdffb640 10 client.4510.objecter ms_dispatch 0x7f2cc8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:29:41.837 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.838+0000 7f2cbdffb640 3 client.4510.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:29:41.837 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.838+0000 7f2cbdffb640 3 client.4510.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:29:41.837 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.838+0000 7f2cbdffb640 20 client.4510.objecter dump_active .. 0 homeless 2024-03-29T11:29:41.837 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.840+0000 7f2ccc835640 10 monclient: get_auth_request con 0x7f2cac06e020 auth_method 0 2024-03-29T11:29:41.838 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.841+0000 7f2cbdffb640 10 client.4510.objecter ms_handle_connect 0x7f2cac06e020 2024-03-29T11:29:41.838 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.845+0000 7f2cbdffb640 1 -- 172.21.15.114:0/15185132 <== mon.1 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2cc8148a20 con 0x7f2cc80a9930 2024-03-29T11:29:41.838 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.845+0000 7f2cbdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:41.838 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:41.845+0000 7f2cbdffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:29:42.140 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:42.148+0000 7f2ccf29a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:42.140 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:42.148+0000 7f2ccf29a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:42.140 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:42.148+0000 7f2ccf29a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:29:42.140 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:42.148+0000 7f2ccf29a640 1 -- 172.21.15.114:0/15185132 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2cc8148e00 con 0x7f2cc80a9930 2024-03-29T11:29:42.140 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:42.149+0000 7f2cbdffb640 1 -- 172.21.15.114:0/15185132 <== mon.1 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2858224613) 0x7f2cc8148e00 con 0x7f2cc80a9930 2024-03-29T11:29:42.140 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:42.149+0000 7f2cbdffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:42.141 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:42.149+0000 7f2cbdffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:29:42.141 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T11:29:42.141 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"b0de2b9b-c45a-44c7-b6ee-e83b39a1e108","created":"2024-03-29T11:20:49.420660+0000","modified":"2024-03-29T11:20:51.700167+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:51.700167+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a64785d3-6b11-499a-b706-cc827ab96a74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5882dadb-e7a8-4604-ba9a-418013c106c8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d7e8c409-bc66-4e4a-bbdb-420bad31388e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0682989-3595-4cd6-90a5-ab9111c278e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"67068120-e3ff-4817-bd06-44feb7479a35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09e8b-8a85-4997-bdea-75d754e6ee67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2f0d9ead-f47f-4cb3-92a4-0a5ad23cd831","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fdb6a25f-6bd8-4c65-9b9b-b460ba7673a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:29:42.146 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:42.152+0000 7f2ccf29a640 10 librados: watch_flush enter 2024-03-29T11:29:42.146 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:42.152+0000 7f2ccf29a640 10 librados: watch_flush exit 2024-03-29T11:29:42.146 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:42.152+0000 7f2ccf29a640 20 client.4510.objecter shutdown clearing up homeless session... 2024-03-29T11:29:42.146 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:42.152+0000 7f2ccf29a640 10 client.4510.objecter successfully canceled tick 2024-03-29T11:29:42.147 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:42.152+0000 7f2ccf29a640 1 -- 172.21.15.114:0/15185132 >> v1:172.21.15.114:6816/4038755055 conn(0x7f2cac06e020 legacy=0x7f2cac0704e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:42.147 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:42.152+0000 7f2ccf29a640 10 monclient: shutdown 2024-03-29T11:29:42.147 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:42.152+0000 7f2ccf29a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:42.147 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:42.152+0000 7f2ccf29a640 1 -- 172.21.15.114:0/15185132 >> v1:172.21.15.176:6789/0 conn(0x7f2cc80a9930 legacy=0x7f2cc81db270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:42.147 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:42.152+0000 7f2ccf29a640 1 -- 172.21.15.114:0/15185132 shutdown_connections 2024-03-29T11:29:42.147 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:42.153+0000 7f2ccf29a640 1 -- 172.21.15.114:0/15185132 >> 172.21.15.114:0/15185132 conn(0x7f2cc80b35e0 msgr2=0x7f2cc80b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:42.148 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:42.153+0000 7f2ccf29a640 1 -- 172.21.15.114:0/15185132 shutdown_connections 2024-03-29T11:29:42.148 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:42.153+0000 7f2ccf29a640 1 -- 172.21.15.114:0/15185132 wait complete. 2024-03-29T11:29:42.148 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:42.153+0000 7f2ccf29a640 1 librados: shutdown 2024-03-29T11:29:42.159 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:29:48.967 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:29:49.116 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.123+0000 7fc9e1e05640 10 monclient: get_monmap_and_config 2024-03-29T11:29:49.116 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.123+0000 7fc9e1e05640 10 monclient: build_initial_monmap 2024-03-29T11:29:49.116 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.123+0000 7fc9e1e05640 10 monclient: monmap: 2024-03-29T11:29:49.116 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:29:49.116 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:29:49.116 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:29:49.124944+0000 2024-03-29T11:29:49.116 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:29:49.124944+0000 2024-03-29T11:29:49.116 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:49.116 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:29:49.116 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:29:49.116 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T11:29:49.117 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T11:29:49.117 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:29:49.117 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.124+0000 7fc9e1e05640 1 Processor -- start 2024-03-29T11:29:49.117 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.125+0000 7fc9e1e05640 1 -- start start 2024-03-29T11:29:49.117 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.125+0000 7fc9e1e05640 10 monclient: init 2024-03-29T11:29:49.119 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.126+0000 7fc9e1e05640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:49.119 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.126+0000 7fc9e1e05640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:29:49.119 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.126+0000 7fc9e1e05640 10 monclient(hunting): picked mon.noname-b con 0x7fc9dc0a9750 addr v1:172.21.15.114:6790/0 2024-03-29T11:29:49.119 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.126+0000 7fc9e1e05640 10 monclient(hunting): picked mon.noname-a con 0x7fc9dc0af2e0 addr v1:172.21.15.114:6789/0 2024-03-29T11:29:49.119 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.126+0000 7fc9e1e05640 10 monclient(hunting): picked mon.noname-c con 0x7fc9dc141ef0 addr v1:172.21.15.176:6789/0 2024-03-29T11:29:49.119 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.126+0000 7fc9e1e05640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9dc0a9b30 con 0x7fc9dc0af2e0 2024-03-29T11:29:49.119 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.127+0000 7fc9e1e05640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9dc0ac4f0 con 0x7fc9dc141ef0 2024-03-29T11:29:49.119 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.127+0000 7fc9e1e05640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9dc0af770 con 0x7fc9dc0a9750 2024-03-29T11:29:49.119 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.127+0000 7fc9e1e05640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:49.120 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.127+0000 7fc9e1e05640 10 monclient(hunting): authenticate will time out at 1459.309290s 2024-03-29T11:29:49.120 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.128+0000 7fc9dbfff640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7fc9dc141ef0 0x7fc9dc1498e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.114:37122/0 (socket says 172.21.15.114:37122) 2024-03-29T11:29:49.120 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.129+0000 7fc9dbfff640 1 -- 172.21.15.114:0/3868324812 learned_addr learned my addr 172.21.15.114:0/3868324812 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:29:49.130 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.131+0000 7fc9da7fc640 1 -- 172.21.15.114:0/3868324812 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1067880100 0 0) 0x7fc9dc0ac4f0 con 0x7fc9dc141ef0 2024-03-29T11:29:49.130 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.131+0000 7fc9da7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:29:49.130 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.131+0000 7fc9da7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:49.130 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.131+0000 7fc9da7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:49.130 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.131+0000 7fc9da7fc640 10 monclient(hunting): my global_id is 4513 2024-03-29T11:29:49.130 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.131+0000 7fc9da7fc640 1 -- 172.21.15.114:0/3868324812 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9c4003520 con 0x7fc9dc141ef0 2024-03-29T11:29:49.130 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.132+0000 7fc9da7fc640 1 -- 172.21.15.114:0/3868324812 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1784906470 0 0) 0x7fc9dc0a9b30 con 0x7fc9dc0af2e0 2024-03-29T11:29:49.130 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.132+0000 7fc9da7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:29:49.130 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.132+0000 7fc9da7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:49.131 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.132+0000 7fc9da7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:49.131 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.132+0000 7fc9da7fc640 10 monclient(hunting): my global_id is 4542 2024-03-29T11:29:49.131 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.132+0000 7fc9da7fc640 1 -- 172.21.15.114:0/3868324812 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9c4003840 con 0x7fc9dc0af2e0 2024-03-29T11:29:49.131 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.132+0000 7fc9da7fc640 1 -- 172.21.15.114:0/3868324812 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2001724846 0 0) 0x7fc9c4003520 con 0x7fc9dc141ef0 2024-03-29T11:29:49.131 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.132+0000 7fc9da7fc640 1 -- 172.21.15.114:0/3868324812 >> v1:172.21.15.114:6790/0 conn(0x7fc9dc0a9750 legacy=0x7fc9dc13b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:29:49.131 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.132+0000 7fc9da7fc640 1 -- 172.21.15.114:0/3868324812 >> v1:172.21.15.114:6789/0 conn(0x7fc9dc0af2e0 legacy=0x7fc9dc1417e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:49.131 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.132+0000 7fc9da7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:29:49.131 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.132+0000 7fc9da7fc640 1 monclient: found mon.noname-c 2024-03-29T11:29:49.131 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.132+0000 7fc9da7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:49.131 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.132+0000 7fc9da7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.176:6789/0 2024-03-29T11:29:49.131 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.132+0000 7fc9da7fc640 1 -- 172.21.15.114:0/3868324812 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9dc0b00c0 con 0x7fc9dc141ef0 2024-03-29T11:29:49.131 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.132+0000 7fc9da7fc640 10 monclient: _finish_auth 0 2024-03-29T11:29:49.131 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.132+0000 7fc9da7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:29:49.132 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.133+0000 7fc9da7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:49.132 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.133+0000 7fc9e1e05640 5 monclient: authenticate success, global_id 4513 2024-03-29T11:29:49.132 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.133+0000 7fc9da7fc640 1 -- 172.21.15.114:0/3868324812 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fc9d40019f0 con 0x7fc9dc141ef0 2024-03-29T11:29:49.132 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.133+0000 7fc9e1e05640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:49.132 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.133+0000 7fc9da7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:49.132 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.133+0000 7fc9da7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:29:49.132 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.133+0000 7fc9da7fc640 10 monclient: dump: 2024-03-29T11:29:49.132 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:29:49.132 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:29:49.132 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:29:49.132 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:29:49.132 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:49.133 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:29:49.133 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:29:49.133 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:29:49.133 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:29:49.133 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:29:49.133 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.133+0000 7fc9e1e05640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:49.133 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.134+0000 7fc9da7fc640 1 -- 172.21.15.114:0/3868324812 <== mon.1 v1:172.21.15.176:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc9d4001bd0 con 0x7fc9dc141ef0 2024-03-29T11:29:49.133 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.134+0000 7fc9da7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:49.133 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.134+0000 7fc9e1e05640 10 monclient: get_monmap_and_config success 2024-03-29T11:29:49.133 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.134+0000 7fc9e1e05640 4 set_mon_vals no callback set 2024-03-29T11:29:49.133 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.134+0000 7fc9da7fc640 1 -- 172.21.15.114:0/3868324812 <== mon.1 v1:172.21.15.176:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fc9d4001f60 con 0x7fc9dc141ef0 2024-03-29T11:29:49.133 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.134+0000 7fc9da7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:49.133 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.134+0000 7fc9da7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:29:49.134 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.134+0000 7fc9da7fc640 10 monclient: dump: 2024-03-29T11:29:49.134 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:29:49.134 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:29:49.134 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:29:49.134 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:29:49.134 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:49.134 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:29:49.135 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:29:49.135 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:29:49.135 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:29:49.135 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:29:49.135 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.135+0000 7fc9e1e05640 10 monclient: shutdown 2024-03-29T11:29:49.135 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.135+0000 7fc9e1e05640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:49.135 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.135+0000 7fc9e1e05640 1 -- 172.21.15.114:0/3868324812 >> v1:172.21.15.176:6789/0 conn(0x7fc9dc141ef0 legacy=0x7fc9dc1498e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:49.135 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.135+0000 7fc9e1e05640 1 -- 172.21.15.114:0/3868324812 shutdown_connections 2024-03-29T11:29:49.135 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.135+0000 7fc9e1e05640 1 -- 172.21.15.114:0/3868324812 >> 172.21.15.114:0/3868324812 conn(0x7fc9dc0b3a00 msgr2=0x7fc9dc0b3de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:49.135 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.135+0000 7fc9e1e05640 1 -- 172.21.15.114:0/3868324812 shutdown_connections 2024-03-29T11:29:49.135 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.135+0000 7fc9e1e05640 1 -- 172.21.15.114:0/3868324812 wait complete. 2024-03-29T11:29:49.136 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.136+0000 7fc9e1e05640 10 monclient: build_initial_monmap 2024-03-29T11:29:49.136 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.136+0000 7fc9e1e05640 10 monclient: monmap: 2024-03-29T11:29:49.136 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:29:49.136 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:29:49.136 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T11:29:49.136 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T11:29:49.136 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:49.136 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:29:49.136 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:29:49.136 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T11:29:49.136 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T11:29:49.137 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:29:49.137 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.137+0000 7fc9e1e05640 1 librados: starting msgr at 2024-03-29T11:29:49.137 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.137+0000 7fc9e1e05640 1 librados: starting objecter 2024-03-29T11:29:49.137 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.137+0000 7fc9e1e05640 1 Processor -- start 2024-03-29T11:29:49.137 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.138+0000 7fc9e1e05640 1 -- start start 2024-03-29T11:29:49.137 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.138+0000 7fc9e1e05640 1 librados: setting wanted keys 2024-03-29T11:29:49.137 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.138+0000 7fc9e1e05640 1 librados: calling monclient init 2024-03-29T11:29:49.137 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.138+0000 7fc9e1e05640 10 monclient: init 2024-03-29T11:29:49.145 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.139+0000 7fc9e1e05640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:49.145 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.139+0000 7fc9e1e05640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:29:49.145 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.140+0000 7fc9e1e05640 10 monclient(hunting): picked mon.noname-c con 0x7fc9dc0a9750 addr v1:172.21.15.114:6790/0 2024-03-29T11:29:49.145 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.140+0000 7fc9e1e05640 10 monclient(hunting): picked mon.noname-b con 0x7fc9dc141ef0 addr v1:172.21.15.176:6789/0 2024-03-29T11:29:49.145 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.140+0000 7fc9e1e05640 10 monclient(hunting): picked mon.noname-a con 0x7fc9dc13c0c0 addr v1:172.21.15.114:6789/0 2024-03-29T11:29:49.145 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.140+0000 7fc9e1e05640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9dc13d230 con 0x7fc9dc13c0c0 2024-03-29T11:29:49.145 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.140+0000 7fc9e1e05640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9dc1d8530 con 0x7fc9dc141ef0 2024-03-29T11:29:49.145 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.140+0000 7fc9e1e05640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9dc13d4a0 con 0x7fc9dc0a9750 2024-03-29T11:29:49.145 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.140+0000 7fc9e1e05640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:49.145 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.140+0000 7fc9e1e05640 10 monclient(hunting): authenticate will time out at 1459.322517s 2024-03-29T11:29:49.145 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.142+0000 7fc9daffd640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7fc9dc141ef0 0x7fc9dc1d7710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.114:37490/0 (socket says 172.21.15.114:37490) 2024-03-29T11:29:49.146 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.142+0000 7fc9daffd640 1 -- 172.21.15.114:0/3515568357 learned_addr learned my addr 172.21.15.114:0/3515568357 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:29:49.146 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.142+0000 7fc9c3fff640 10 client.?.objecter ms_handle_connect 0x7fc9dc141ef0 2024-03-29T11:29:49.146 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.143+0000 7fc9c3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:49.146 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.143+0000 7fc9c3fff640 1 -- 172.21.15.114:0/3515568357 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2082370872 0 0) 0x7fc9dc1d8530 con 0x7fc9dc141ef0 2024-03-29T11:29:49.146 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.143+0000 7fc9c3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:29:49.146 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.143+0000 7fc9c3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:49.146 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.143+0000 7fc9c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:49.146 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.143+0000 7fc9c3fff640 10 monclient(hunting): my global_id is 4516 2024-03-29T11:29:49.146 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.143+0000 7fc9c3fff640 1 -- 172.21.15.114:0/3515568357 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9c8004eb0 con 0x7fc9dc141ef0 2024-03-29T11:29:49.146 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.143+0000 7fc9c3fff640 10 client.?.objecter ms_handle_connect 0x7fc9dc0a9750 2024-03-29T11:29:49.146 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.143+0000 7fc9c3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:49.147 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.144+0000 7fc9c3fff640 1 -- 172.21.15.114:0/3515568357 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3476613597 0 0) 0x7fc9dc13d4a0 con 0x7fc9dc0a9750 2024-03-29T11:29:49.147 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.144+0000 7fc9c3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:29:49.147 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.144+0000 7fc9c3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:49.147 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.144+0000 7fc9c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:49.147 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.144+0000 7fc9c3fff640 10 monclient(hunting): my global_id is 4574 2024-03-29T11:29:49.147 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.144+0000 7fc9c3fff640 1 -- 172.21.15.114:0/3515568357 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9dc1d8530 con 0x7fc9dc0a9750 2024-03-29T11:29:49.147 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.144+0000 7fc9c3fff640 1 -- 172.21.15.114:0/3515568357 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 983037627 0 0) 0x7fc9c8004eb0 con 0x7fc9dc141ef0 2024-03-29T11:29:49.147 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.144+0000 7fc9c3fff640 1 -- 172.21.15.114:0/3515568357 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc9dc13d4a0 con 0x7fc9dc141ef0 2024-03-29T11:29:49.147 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.144+0000 7fc9c3fff640 1 -- 172.21.15.114:0/3515568357 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fc9d0001990 con 0x7fc9dc141ef0 2024-03-29T11:29:49.147 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.144+0000 7fc9c3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:49.147 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.144+0000 7fc9c3fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:29:49.147 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.144+0000 7fc9c3fff640 10 monclient(hunting): dump: 2024-03-29T11:29:49.147 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:29:49.147 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:29:49.148 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:29:49.148 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:29:49.148 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:49.148 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:29:49.148 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:29:49.148 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:29:49.148 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:29:49.148 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:29:49.148 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.144+0000 7fc9c3fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:29:49.148 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.145+0000 7fc9c3fff640 1 -- 172.21.15.114:0/3515568357 <== mon.1 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2189080374 0 0) 0x7fc9dc13d4a0 con 0x7fc9dc141ef0 2024-03-29T11:29:49.148 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.145+0000 7fc9c3fff640 1 -- 172.21.15.114:0/3515568357 >> v1:172.21.15.114:6790/0 conn(0x7fc9dc0a9750 legacy=0x7fc9dc13d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:49.148 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.145+0000 7fc9c3fff640 1 -- 172.21.15.114:0/3515568357 >> v1:172.21.15.114:6789/0 conn(0x7fc9dc13c0c0 legacy=0x7fc9dc1d7e20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:29:49.148 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.146+0000 7fc9c3fff640 10 monclient: _finish_hunting 0 2024-03-29T11:29:49.148 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.147+0000 7fc9c3fff640 1 monclient: found mon.b 2024-03-29T11:29:49.148 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.147+0000 7fc9c3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:49.149 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.147+0000 7fc9c3fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:29:49.149 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.147+0000 7fc9c3fff640 1 -- 172.21.15.114:0/3515568357 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9dc12e3a0 con 0x7fc9dc141ef0 2024-03-29T11:29:49.149 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.147+0000 7fc9c3fff640 10 monclient: _finish_auth 0 2024-03-29T11:29:49.149 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.147+0000 7fc9c3fff640 10 monclient: _check_auth_tickets 2024-03-29T11:29:49.149 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.147+0000 7fc9c3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:49.149 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.147+0000 7fc9e1e05640 5 monclient: authenticate success, global_id 4516 2024-03-29T11:29:49.149 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.147+0000 7fc9e1e05640 10 monclient: _renew_subs 2024-03-29T11:29:49.150 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.147+0000 7fc9e1e05640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:29:49.150 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.147+0000 7fc9e1e05640 1 -- 172.21.15.114:0/3515568357 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9dc12e070 con 0x7fc9dc141ef0 2024-03-29T11:29:49.150 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.147+0000 7fc9e1e05640 10 client.4516.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:29:49.150 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.147+0000 7fc9c3fff640 1 -- 172.21.15.114:0/3515568357 <== mon.1 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc9d0001af0 con 0x7fc9dc141ef0 2024-03-29T11:29:49.150 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.147+0000 7fc9c3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:49.150 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.147+0000 7fc9c3fff640 1 -- 172.21.15.114:0/3515568357 <== mon.1 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fc9d0003a60 con 0x7fc9dc141ef0 2024-03-29T11:29:49.150 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.147+0000 7fc9d9ffb640 4 set_mon_vals no callback set 2024-03-29T11:29:49.150 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.147+0000 7fc9c3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:49.150 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.147+0000 7fc9c3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:29:49.150 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.147+0000 7fc9c3fff640 10 monclient: dump: 2024-03-29T11:29:49.150 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:29:49.150 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:29:49.150 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:29:49.150 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:29:49.151 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:49.151 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:29:49.151 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:29:49.151 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:29:49.151 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:29:49.151 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:29:49.151 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.147+0000 7fc9e1e05640 10 monclient: _renew_subs 2024-03-29T11:29:49.151 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.147+0000 7fc9e1e05640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:29:49.151 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.147+0000 7fc9e1e05640 1 -- 172.21.15.114:0/3515568357 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9dc12d5d0 con 0x7fc9dc141ef0 2024-03-29T11:29:49.151 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.147+0000 7fc9e1e05640 1 librados: init done 2024-03-29T11:29:49.151 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.149+0000 7fc9c3fff640 1 -- 172.21.15.114:0/3515568357 <== mon.1 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1594292461 0 0) 0x7fc9d0003bc0 con 0x7fc9dc141ef0 2024-03-29T11:29:49.151 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.149+0000 7fc9e1e05640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:49.151 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.149+0000 7fc9e1e05640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:49.151 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.149+0000 7fc9e1e05640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:29:49.152 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.149+0000 7fc9e1e05640 1 -- 172.21.15.114:0/3515568357 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9dc12d920 con 0x7fc9dc141ef0 2024-03-29T11:29:49.152 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.150+0000 7fc9db7fe640 10 monclient: get_auth_request con 0x7fc9c806dba0 auth_method 0 2024-03-29T11:29:49.152 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.150+0000 7fc9c3fff640 10 client.4516.objecter ms_handle_connect 0x7fc9c806dba0 2024-03-29T11:29:49.152 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.151+0000 7fc9c3fff640 1 -- 172.21.15.114:0/3515568357 <== mon.1 v1:172.21.15.176:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3238987313 0 0) 0x7fc9d0088f20 con 0x7fc9dc141ef0 2024-03-29T11:29:49.152 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.151+0000 7fc9c3fff640 10 client.4516.objecter ms_dispatch 0x7fc9dc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:29:49.152 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.151+0000 7fc9c3fff640 3 client.4516.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:29:49.152 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.151+0000 7fc9c3fff640 3 client.4516.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:29:49.152 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.151+0000 7fc9c3fff640 20 client.4516.objecter dump_active .. 0 homeless 2024-03-29T11:29:49.153 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.160+0000 7fc9c3fff640 1 -- 172.21.15.114:0/3515568357 <== mon.1 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc9d0054840 con 0x7fc9dc141ef0 2024-03-29T11:29:49.153 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.160+0000 7fc9c3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:49.153 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.160+0000 7fc9c3fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:29:49.484 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.492+0000 7fc9e1e05640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:49.484 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.492+0000 7fc9e1e05640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:49.484 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.492+0000 7fc9e1e05640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:29:49.484 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.492+0000 7fc9e1e05640 1 -- 172.21.15.114:0/3515568357 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc9dc04f730 con 0x7fc9dc141ef0 2024-03-29T11:29:49.489 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.497+0000 7fc9c3fff640 1 -- 172.21.15.114:0/3515568357 <== mon.1 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2858224613) 0x7fc9d0056770 con 0x7fc9dc141ef0 2024-03-29T11:29:49.489 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.497+0000 7fc9c3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:49.489 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.497+0000 7fc9c3fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:29:49.490 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T11:29:49.490 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"b0de2b9b-c45a-44c7-b6ee-e83b39a1e108","created":"2024-03-29T11:20:49.420660+0000","modified":"2024-03-29T11:20:51.700167+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:51.700167+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a64785d3-6b11-499a-b706-cc827ab96a74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5882dadb-e7a8-4604-ba9a-418013c106c8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d7e8c409-bc66-4e4a-bbdb-420bad31388e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0682989-3595-4cd6-90a5-ab9111c278e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"67068120-e3ff-4817-bd06-44feb7479a35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09e8b-8a85-4997-bdea-75d754e6ee67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2f0d9ead-f47f-4cb3-92a4-0a5ad23cd831","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fdb6a25f-6bd8-4c65-9b9b-b460ba7673a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:29:49.494 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.501+0000 7fc9e1e05640 10 librados: watch_flush enter 2024-03-29T11:29:49.494 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.501+0000 7fc9e1e05640 10 librados: watch_flush exit 2024-03-29T11:29:49.494 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.501+0000 7fc9e1e05640 20 client.4516.objecter shutdown clearing up homeless session... 2024-03-29T11:29:49.494 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.501+0000 7fc9e1e05640 10 client.4516.objecter successfully canceled tick 2024-03-29T11:29:49.494 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.501+0000 7fc9e1e05640 1 -- 172.21.15.114:0/3515568357 >> v1:172.21.15.114:6816/4038755055 conn(0x7fc9c806dba0 legacy=0x7fc9c8070060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:49.495 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.501+0000 7fc9e1e05640 10 monclient: shutdown 2024-03-29T11:29:49.495 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.501+0000 7fc9e1e05640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:49.495 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.501+0000 7fc9e1e05640 1 -- 172.21.15.114:0/3515568357 >> v1:172.21.15.176:6789/0 conn(0x7fc9dc141ef0 legacy=0x7fc9dc1d7710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:49.495 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.502+0000 7fc9e1e05640 1 -- 172.21.15.114:0/3515568357 shutdown_connections 2024-03-29T11:29:49.495 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.502+0000 7fc9e1e05640 1 -- 172.21.15.114:0/3515568357 >> 172.21.15.114:0/3515568357 conn(0x7fc9dc0b3a00 msgr2=0x7fc9dc0b3de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:49.496 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.504+0000 7fc9e1e05640 1 -- 172.21.15.114:0/3515568357 shutdown_connections 2024-03-29T11:29:49.496 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.504+0000 7fc9e1e05640 1 -- 172.21.15.114:0/3515568357 wait complete. 2024-03-29T11:29:49.496 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:49.504+0000 7fc9e1e05640 1 librados: shutdown 2024-03-29T11:29:49.512 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:29:56.319 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:29:56.467 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.475+0000 7fa57704a640 10 monclient: get_monmap_and_config 2024-03-29T11:29:56.467 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.475+0000 7fa57704a640 10 monclient: build_initial_monmap 2024-03-29T11:29:56.467 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.475+0000 7fa57704a640 10 monclient: monmap: 2024-03-29T11:29:56.468 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:29:56.468 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:29:56.468 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:29:56.476451+0000 2024-03-29T11:29:56.468 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:29:56.476451+0000 2024-03-29T11:29:56.468 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:56.468 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:29:56.468 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:29:56.468 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T11:29:56.468 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T11:29:56.468 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:29:56.469 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.476+0000 7fa57704a640 1 Processor -- start 2024-03-29T11:29:56.469 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.476+0000 7fa57704a640 1 -- start start 2024-03-29T11:29:56.469 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.476+0000 7fa57704a640 10 monclient: init 2024-03-29T11:29:56.469 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.477+0000 7fa57704a640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:56.469 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.477+0000 7fa57704a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:29:56.469 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.477+0000 7fa57704a640 10 monclient(hunting): picked mon.noname-b con 0x7fa5700b45a0 addr v1:172.21.15.114:6790/0 2024-03-29T11:29:56.469 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.477+0000 7fa57704a640 10 monclient(hunting): picked mon.noname-c con 0x7fa57013fb10 addr v1:172.21.15.176:6789/0 2024-03-29T11:29:56.470 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.477+0000 7fa57704a640 10 monclient(hunting): picked mon.noname-a con 0x7fa5700b3920 addr v1:172.21.15.114:6789/0 2024-03-29T11:29:56.470 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.477+0000 7fa57704a640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5700b1de0 con 0x7fa5700b3920 2024-03-29T11:29:56.470 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.477+0000 7fa57704a640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5700b0670 con 0x7fa57013fb10 2024-03-29T11:29:56.470 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.477+0000 7fa57704a640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5700af560 con 0x7fa5700b45a0 2024-03-29T11:29:56.470 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.477+0000 7fa57704a640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:56.470 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.477+0000 7fa57704a640 10 monclient(hunting): authenticate will time out at 1466.659741s 2024-03-29T11:29:56.479 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.482+0000 7fa56ffff640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7fa57013fb10 0x7fa5700b0fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.114:55277/0 (socket says 172.21.15.114:55277) 2024-03-29T11:29:56.479 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.482+0000 7fa56ffff640 1 -- 172.21.15.114:0/3352454940 learned_addr learned my addr 172.21.15.114:0/3352454940 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:29:56.479 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.482+0000 7fa56f7fe640 1 -- 172.21.15.114:0/3352454940 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2871587623 0 0) 0x7fa5700b0670 con 0x7fa57013fb10 2024-03-29T11:29:56.479 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.482+0000 7fa56f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:29:56.479 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.482+0000 7fa56f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:56.479 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.482+0000 7fa56f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:56.479 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.482+0000 7fa56f7fe640 10 monclient(hunting): my global_id is 4519 2024-03-29T11:29:56.479 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.482+0000 7fa56f7fe640 1 -- 172.21.15.114:0/3352454940 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa558003520 con 0x7fa57013fb10 2024-03-29T11:29:56.479 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.482+0000 7fa56f7fe640 1 -- 172.21.15.114:0/3352454940 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1994869945 0 0) 0x7fa558003520 con 0x7fa57013fb10 2024-03-29T11:29:56.479 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.482+0000 7fa56f7fe640 1 -- 172.21.15.114:0/3352454940 >> v1:172.21.15.114:6790/0 conn(0x7fa5700b45a0 legacy=0x7fa57013f400 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:29:56.480 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.484+0000 7fa56f7fe640 1 -- 172.21.15.114:0/3352454940 >> v1:172.21.15.114:6789/0 conn(0x7fa5700b3920 legacy=0x7fa5700b16d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:29:56.480 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.484+0000 7fa56f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:29:56.480 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.484+0000 7fa56f7fe640 1 monclient: found mon.noname-c 2024-03-29T11:29:56.480 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.484+0000 7fa56f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:56.480 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.484+0000 7fa56f7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.176:6789/0 2024-03-29T11:29:56.480 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.484+0000 7fa56f7fe640 1 -- 172.21.15.114:0/3352454940 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5700b08c0 con 0x7fa57013fb10 2024-03-29T11:29:56.480 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.484+0000 7fa56f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:29:56.480 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.484+0000 7fa56f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:29:56.480 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.484+0000 7fa56f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:56.480 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.484+0000 7fa57704a640 5 monclient: authenticate success, global_id 4519 2024-03-29T11:29:56.480 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.484+0000 7fa57704a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:56.480 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.484+0000 7fa56f7fe640 1 -- 172.21.15.114:0/3352454940 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fa564001690 con 0x7fa57013fb10 2024-03-29T11:29:56.480 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.484+0000 7fa56f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:56.480 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.484+0000 7fa56f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:29:56.481 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.484+0000 7fa56f7fe640 10 monclient: dump: 2024-03-29T11:29:56.481 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:29:56.481 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:29:56.481 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:29:56.481 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:29:56.481 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:56.481 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:29:56.481 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:29:56.481 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:29:56.481 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:29:56.481 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:29:56.481 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.484+0000 7fa57704a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:56.481 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.485+0000 7fa56f7fe640 1 -- 172.21.15.114:0/3352454940 <== mon.1 v1:172.21.15.176:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa564001920 con 0x7fa57013fb10 2024-03-29T11:29:56.481 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.485+0000 7fa56f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:56.482 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.485+0000 7fa57704a640 10 monclient: get_monmap_and_config success 2024-03-29T11:29:56.482 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.485+0000 7fa57704a640 4 set_mon_vals no callback set 2024-03-29T11:29:56.482 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.485+0000 7fa56f7fe640 1 -- 172.21.15.114:0/3352454940 <== mon.1 v1:172.21.15.176:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fa564001cb0 con 0x7fa57013fb10 2024-03-29T11:29:56.482 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.485+0000 7fa56f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:56.482 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.485+0000 7fa56f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:29:56.482 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.485+0000 7fa56f7fe640 10 monclient: dump: 2024-03-29T11:29:56.482 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:29:56.482 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:29:56.482 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:29:56.482 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:29:56.482 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:56.482 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:29:56.482 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:29:56.482 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:29:56.483 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:29:56.483 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:29:56.483 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.486+0000 7fa57704a640 10 monclient: shutdown 2024-03-29T11:29:56.483 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.486+0000 7fa57704a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:56.483 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.486+0000 7fa57704a640 1 -- 172.21.15.114:0/3352454940 >> v1:172.21.15.176:6789/0 conn(0x7fa57013fb10 legacy=0x7fa5700b0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:56.483 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.486+0000 7fa57704a640 1 -- 172.21.15.114:0/3352454940 shutdown_connections 2024-03-29T11:29:56.483 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.486+0000 7fa57704a640 1 -- 172.21.15.114:0/3352454940 >> 172.21.15.114:0/3352454940 conn(0x7fa570066a00 msgr2=0x7fa570066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:56.483 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.486+0000 7fa57704a640 1 -- 172.21.15.114:0/3352454940 shutdown_connections 2024-03-29T11:29:56.483 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.486+0000 7fa57704a640 1 -- 172.21.15.114:0/3352454940 wait complete. 2024-03-29T11:29:56.484 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.487+0000 7fa57704a640 10 monclient: build_initial_monmap 2024-03-29T11:29:56.484 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.487+0000 7fa57704a640 10 monclient: monmap: 2024-03-29T11:29:56.484 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:29:56.484 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:29:56.484 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T11:29:56.484 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T11:29:56.484 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:56.484 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:29:56.484 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:29:56.484 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T11:29:56.484 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T11:29:56.484 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:29:56.485 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.488+0000 7fa57704a640 1 librados: starting msgr at 2024-03-29T11:29:56.485 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.488+0000 7fa57704a640 1 librados: starting objecter 2024-03-29T11:29:56.485 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.488+0000 7fa57704a640 1 Processor -- start 2024-03-29T11:29:56.485 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.488+0000 7fa57704a640 1 -- start start 2024-03-29T11:29:56.485 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.488+0000 7fa57704a640 1 librados: setting wanted keys 2024-03-29T11:29:56.485 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.488+0000 7fa57704a640 1 librados: calling monclient init 2024-03-29T11:29:56.485 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.488+0000 7fa57704a640 10 monclient: init 2024-03-29T11:29:56.485 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.489+0000 7fa57704a640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:56.485 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.489+0000 7fa57704a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:29:56.485 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.489+0000 7fa57704a640 10 monclient(hunting): picked mon.noname-c con 0x7fa5700b3920 addr v1:172.21.15.114:6790/0 2024-03-29T11:29:56.485 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.489+0000 7fa57704a640 10 monclient(hunting): picked mon.noname-b con 0x7fa5700b45a0 addr v1:172.21.15.176:6789/0 2024-03-29T11:29:56.485 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.490+0000 7fa574de6640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7fa5700b3920 0x7fa5701e3c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:34830/0 (socket says 172.21.15.114:34830) 2024-03-29T11:29:56.486 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.492+0000 7fa57704a640 10 monclient(hunting): picked mon.noname-a con 0x7fa57013fb10 addr v1:172.21.15.114:6789/0 2024-03-29T11:29:56.486 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.492+0000 7fa57704a640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5701e39b0 con 0x7fa57013fb10 2024-03-29T11:29:56.486 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.492+0000 7fa57704a640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5700a9d80 con 0x7fa5700b45a0 2024-03-29T11:29:56.486 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.492+0000 7fa57704a640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5700aa060 con 0x7fa5700b3920 2024-03-29T11:29:56.486 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.492+0000 7fa57704a640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:56.486 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.492+0000 7fa57704a640 10 monclient(hunting): authenticate will time out at 1466.674985s 2024-03-29T11:29:56.489 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.495+0000 7fa574de6640 1 -- 172.21.15.114:0/717856139 learned_addr learned my addr 172.21.15.114:0/717856139 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:29:56.490 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.495+0000 7fa5755e7640 1 --1- 172.21.15.114:0/717856139 >> v1:172.21.15.114:6789/0 conn(0x7fa57013fb10 0x7fa5701e32a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:32859/0 (socket says 172.21.15.114:32859) 2024-03-29T11:29:56.490 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.495+0000 7fa56d7fa640 10 client.?.objecter ms_handle_connect 0x7fa5700b3920 2024-03-29T11:29:56.490 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.495+0000 7fa56d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:56.490 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.495+0000 7fa56d7fa640 10 client.?.objecter ms_handle_connect 0x7fa57013fb10 2024-03-29T11:29:56.490 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.495+0000 7fa56d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:56.490 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.495+0000 7fa56d7fa640 1 -- 172.21.15.114:0/717856139 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 773723728 0 0) 0x7fa5700aa060 con 0x7fa5700b3920 2024-03-29T11:29:56.490 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.495+0000 7fa56d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:29:56.490 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.495+0000 7fa56d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:56.490 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.495+0000 7fa56d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:56.490 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.495+0000 7fa56d7fa640 10 monclient(hunting): my global_id is 4577 2024-03-29T11:29:56.490 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.495+0000 7fa56d7fa640 1 -- 172.21.15.114:0/717856139 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa55c004eb0 con 0x7fa5700b3920 2024-03-29T11:29:56.490 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.495+0000 7fa56d7fa640 1 -- 172.21.15.114:0/717856139 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2139790612 0 0) 0x7fa5701e39b0 con 0x7fa57013fb10 2024-03-29T11:29:56.490 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.495+0000 7fa56d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:29:56.491 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.495+0000 7fa56d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:56.491 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.495+0000 7fa56d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:56.491 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.495+0000 7fa56d7fa640 10 monclient(hunting): my global_id is 4545 2024-03-29T11:29:56.491 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.495+0000 7fa56d7fa640 1 -- 172.21.15.114:0/717856139 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5700aa060 con 0x7fa57013fb10 2024-03-29T11:29:56.491 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.496+0000 7fa56d7fa640 1 -- 172.21.15.114:0/717856139 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3878026793 0 0) 0x7fa55c004eb0 con 0x7fa5700b3920 2024-03-29T11:29:56.491 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.496+0000 7fa56d7fa640 1 -- 172.21.15.114:0/717856139 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa5701e39b0 con 0x7fa5700b3920 2024-03-29T11:29:56.491 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.496+0000 7fa56d7fa640 1 -- 172.21.15.114:0/717856139 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fa568001960 con 0x7fa5700b3920 2024-03-29T11:29:56.491 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.496+0000 7fa56d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:56.491 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.496+0000 7fa56d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:29:56.491 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.496+0000 7fa56d7fa640 10 monclient(hunting): dump: 2024-03-29T11:29:56.491 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:29:56.491 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:29:56.491 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:29:56.492 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:29:56.492 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:56.492 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:29:56.492 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:29:56.492 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:29:56.492 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:29:56.492 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:29:56.492 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.496+0000 7fa56d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:29:56.492 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.496+0000 7fa56d7fa640 1 -- 172.21.15.114:0/717856139 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3946947680 0 0) 0x7fa5700aa060 con 0x7fa57013fb10 2024-03-29T11:29:56.492 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.496+0000 7fa56d7fa640 1 -- 172.21.15.114:0/717856139 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa55c004eb0 con 0x7fa57013fb10 2024-03-29T11:29:56.492 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.496+0000 7fa56d7fa640 1 -- 172.21.15.114:0/717856139 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fa560001a90 con 0x7fa57013fb10 2024-03-29T11:29:56.492 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.496+0000 7fa56d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:56.492 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.496+0000 7fa56d7fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:29:56.492 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.496+0000 7fa56d7fa640 10 monclient(hunting): dump: 2024-03-29T11:29:56.492 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:29:56.493 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:29:56.493 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:29:56.493 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:29:56.493 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:56.493 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:29:56.493 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:29:56.493 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:29:56.493 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:29:56.493 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:29:56.493 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.496+0000 7fa56d7fa640 1 -- 172.21.15.114:0/717856139 <== mon.2 v1:172.21.15.114:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2833743055 0 0) 0x7fa5701e39b0 con 0x7fa5700b3920 2024-03-29T11:29:56.493 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.496+0000 7fa56d7fa640 1 -- 172.21.15.114:0/717856139 >> v1:172.21.15.176:6789/0 conn(0x7fa5700b45a0 legacy=0x7fa5701e2b90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:29:56.495 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.498+0000 7fa56d7fa640 1 -- 172.21.15.114:0/717856139 >> v1:172.21.15.114:6789/0 conn(0x7fa57013fb10 legacy=0x7fa5701e32a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:56.495 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.498+0000 7fa56d7fa640 10 monclient: _finish_hunting 0 2024-03-29T11:29:56.495 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.498+0000 7fa56d7fa640 1 monclient: found mon.c 2024-03-29T11:29:56.495 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.498+0000 7fa56d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:56.495 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.498+0000 7fa56d7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:29:56.495 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.498+0000 7fa56d7fa640 1 -- 172.21.15.114:0/717856139 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa570071690 con 0x7fa5700b3920 2024-03-29T11:29:56.495 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.498+0000 7fa56d7fa640 10 monclient: _finish_auth 0 2024-03-29T11:29:56.496 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.498+0000 7fa56d7fa640 10 monclient: _check_auth_tickets 2024-03-29T11:29:56.496 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.498+0000 7fa56d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:56.496 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.498+0000 7fa56d7fa640 10 client.?.objecter ms_handle_connect 0x7fa5700b45a0 2024-03-29T11:29:56.496 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.498+0000 7fa56d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:56.496 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.498+0000 7fa57704a640 5 monclient: authenticate success, global_id 4577 2024-03-29T11:29:56.496 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.498+0000 7fa57704a640 10 monclient: _renew_subs 2024-03-29T11:29:56.496 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.498+0000 7fa57704a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:29:56.496 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.498+0000 7fa57704a640 1 -- 172.21.15.114:0/717856139 --> v1:172.21.15.114:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa570070860 con 0x7fa5700b3920 2024-03-29T11:29:56.496 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.498+0000 7fa57704a640 10 client.4577.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:29:56.496 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.498+0000 7fa57704a640 10 monclient: _renew_subs 2024-03-29T11:29:56.496 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.498+0000 7fa57704a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:29:56.496 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.498+0000 7fa57704a640 1 -- 172.21.15.114:0/717856139 --> v1:172.21.15.114:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa570071000 con 0x7fa5700b3920 2024-03-29T11:29:56.496 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.498+0000 7fa57704a640 1 librados: init done 2024-03-29T11:29:56.496 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.498+0000 7fa56d7fa640 1 -- 172.21.15.114:0/717856139 <== mon.2 v1:172.21.15.114:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa5680035f0 con 0x7fa5700b3920 2024-03-29T11:29:56.497 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.498+0000 7fa56d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:56.497 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.498+0000 7fa56e7fc640 4 set_mon_vals no callback set 2024-03-29T11:29:56.497 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.499+0000 7fa56d7fa640 1 -- 172.21.15.114:0/717856139 <== mon.2 v1:172.21.15.114:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fa568003940 con 0x7fa5700b3920 2024-03-29T11:29:56.497 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.499+0000 7fa56d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:56.497 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.499+0000 7fa56d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:29:56.497 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.499+0000 7fa56d7fa640 10 monclient: dump: 2024-03-29T11:29:56.497 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:29:56.497 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:29:56.497 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:29:56.497 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:29:56.497 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:56.497 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:29:56.497 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:29:56.497 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:29:56.498 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:29:56.498 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:29:56.498 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.499+0000 7fa57704a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:56.498 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.499+0000 7fa57704a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:56.498 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.499+0000 7fa57704a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:29:56.498 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.499+0000 7fa57704a640 1 -- 172.21.15.114:0/717856139 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa550005080 con 0x7fa5700b3920 2024-03-29T11:29:56.498 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.501+0000 7fa56d7fa640 1 -- 172.21.15.114:0/717856139 <== mon.2 v1:172.21.15.114:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1594292461 0 0) 0x7fa568003aa0 con 0x7fa5700b3920 2024-03-29T11:29:56.498 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.501+0000 7fa56d7fa640 1 -- 172.21.15.114:0/717856139 <== mon.2 v1:172.21.15.114:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3238987313 0 0) 0x7fa568084980 con 0x7fa5700b3920 2024-03-29T11:29:56.498 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.501+0000 7fa56d7fa640 10 client.4577.objecter ms_dispatch 0x7fa570000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:29:56.498 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.501+0000 7fa56d7fa640 3 client.4577.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:29:56.498 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.501+0000 7fa56d7fa640 3 client.4577.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:29:56.498 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.501+0000 7fa56d7fa640 20 client.4577.objecter dump_active .. 0 homeless 2024-03-29T11:29:56.498 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.502+0000 7fa56ffff640 10 monclient: get_auth_request con 0x7fa55c06ded0 auth_method 0 2024-03-29T11:29:56.498 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.503+0000 7fa56d7fa640 10 client.4577.objecter ms_handle_connect 0x7fa55c06ded0 2024-03-29T11:29:56.503 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.511+0000 7fa56d7fa640 1 -- 172.21.15.114:0/717856139 <== mon.2 v1:172.21.15.114:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa568050260 con 0x7fa5700b3920 2024-03-29T11:29:56.503 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.511+0000 7fa56d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:56.503 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.511+0000 7fa56d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:29:56.831 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.839+0000 7fa57704a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:56.832 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.839+0000 7fa57704a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:56.832 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.839+0000 7fa57704a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:29:56.832 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.839+0000 7fa57704a640 1 -- 172.21.15.114:0/717856139 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa550005330 con 0x7fa5700b3920 2024-03-29T11:29:56.833 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.841+0000 7fa56d7fa640 1 -- 172.21.15.114:0/717856139 <== mon.2 v1:172.21.15.114:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2858224613) 0x7fa568052190 con 0x7fa5700b3920 2024-03-29T11:29:56.833 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.841+0000 7fa56d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:56.833 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.841+0000 7fa56d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:29:56.834 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T11:29:56.834 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"b0de2b9b-c45a-44c7-b6ee-e83b39a1e108","created":"2024-03-29T11:20:49.420660+0000","modified":"2024-03-29T11:20:51.700167+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:51.700167+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a64785d3-6b11-499a-b706-cc827ab96a74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5882dadb-e7a8-4604-ba9a-418013c106c8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d7e8c409-bc66-4e4a-bbdb-420bad31388e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0682989-3595-4cd6-90a5-ab9111c278e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"67068120-e3ff-4817-bd06-44feb7479a35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09e8b-8a85-4997-bdea-75d754e6ee67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2f0d9ead-f47f-4cb3-92a4-0a5ad23cd831","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fdb6a25f-6bd8-4c65-9b9b-b460ba7673a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:29:56.837 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.845+0000 7fa57704a640 10 librados: watch_flush enter 2024-03-29T11:29:56.838 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.845+0000 7fa57704a640 10 librados: watch_flush exit 2024-03-29T11:29:56.838 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.845+0000 7fa57704a640 20 client.4577.objecter shutdown clearing up homeless session... 2024-03-29T11:29:56.839 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.845+0000 7fa57704a640 10 client.4577.objecter successfully canceled tick 2024-03-29T11:29:56.839 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.845+0000 7fa57704a640 1 -- 172.21.15.114:0/717856139 >> v1:172.21.15.114:6816/4038755055 conn(0x7fa55c06ded0 legacy=0x7fa55c070390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:56.839 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.845+0000 7fa57704a640 10 monclient: shutdown 2024-03-29T11:29:56.839 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.845+0000 7fa57704a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:56.839 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.845+0000 7fa57704a640 1 -- 172.21.15.114:0/717856139 >> v1:172.21.15.114:6790/0 conn(0x7fa5700b3920 legacy=0x7fa5701e3c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:56.839 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.846+0000 7fa57704a640 1 -- 172.21.15.114:0/717856139 shutdown_connections 2024-03-29T11:29:56.839 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.846+0000 7fa57704a640 1 -- 172.21.15.114:0/717856139 >> 172.21.15.114:0/717856139 conn(0x7fa570066a00 msgr2=0x7fa570139aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:56.840 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.848+0000 7fa57704a640 1 -- 172.21.15.114:0/717856139 shutdown_connections 2024-03-29T11:29:56.840 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.848+0000 7fa57704a640 1 -- 172.21.15.114:0/717856139 wait complete. 2024-03-29T11:29:56.840 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:29:56.848+0000 7fa57704a640 1 librados: shutdown 2024-03-29T11:29:56.857 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:30:03.664 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:30:03.814 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.822+0000 7f7fae169640 10 monclient: get_monmap_and_config 2024-03-29T11:30:03.814 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.822+0000 7f7fae169640 10 monclient: build_initial_monmap 2024-03-29T11:30:03.815 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.822+0000 7f7fae169640 10 monclient: monmap: 2024-03-29T11:30:03.815 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:30:03.815 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:30:03.815 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:30:03.823425+0000 2024-03-29T11:30:03.815 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:30:03.823425+0000 2024-03-29T11:30:03.815 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:03.815 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:30:03.815 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:30:03.815 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T11:30:03.815 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T11:30:03.815 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:30:03.816 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.823+0000 7f7fae169640 1 Processor -- start 2024-03-29T11:30:03.816 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.823+0000 7f7fae169640 1 -- start start 2024-03-29T11:30:03.816 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.823+0000 7f7fae169640 10 monclient: init 2024-03-29T11:30:03.817 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.824+0000 7f7fae169640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:03.817 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.824+0000 7f7fae169640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:30:03.817 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.824+0000 7f7fae169640 10 monclient(hunting): picked mon.noname-c con 0x7f7fa0036a60 addr v1:172.21.15.176:6789/0 2024-03-29T11:30:03.817 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.824+0000 7f7fae169640 10 monclient(hunting): picked mon.noname-b con 0x7f7fa00c5de0 addr v1:172.21.15.114:6790/0 2024-03-29T11:30:03.817 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.824+0000 7f7fae169640 10 monclient(hunting): picked mon.noname-a con 0x7f7fa00c3070 addr v1:172.21.15.114:6789/0 2024-03-29T11:30:03.817 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.824+0000 7f7fae169640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fa002cc60 con 0x7f7fa00c3070 2024-03-29T11:30:03.817 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.824+0000 7f7fae169640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fa002e040 con 0x7f7fa0036a60 2024-03-29T11:30:03.817 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.824+0000 7f7fae169640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fa002e320 con 0x7f7fa00c5de0 2024-03-29T11:30:03.817 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.824+0000 7f7fae169640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:03.817 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.824+0000 7f7fae169640 10 monclient(hunting): authenticate will time out at 1474.006976s 2024-03-29T11:30:03.817 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.825+0000 7f7fac966640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f7fa00c5de0 0x7f7fa00c2960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:50322/0 (socket says 172.21.15.114:50322) 2024-03-29T11:30:03.818 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.825+0000 7f7fac966640 1 -- 172.21.15.114:0/2658068377 learned_addr learned my addr 172.21.15.114:0/2658068377 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:30:03.822 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.828+0000 7f7f9ffff640 1 -- 172.21.15.114:0/2658068377 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3233607116 0 0) 0x7f7fa002e320 con 0x7f7fa00c5de0 2024-03-29T11:30:03.822 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.828+0000 7f7f9ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:30:03.823 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.828+0000 7f7f9ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:03.823 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.828+0000 7f7f9ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:03.823 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.828+0000 7f7f9ffff640 10 monclient(hunting): my global_id is 4580 2024-03-29T11:30:03.823 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.828+0000 7f7f9ffff640 1 -- 172.21.15.114:0/2658068377 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f94003520 con 0x7f7fa00c5de0 2024-03-29T11:30:03.823 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.828+0000 7f7f9ffff640 1 -- 172.21.15.114:0/2658068377 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1434734060 0 0) 0x7f7fa002cc60 con 0x7f7fa00c3070 2024-03-29T11:30:03.823 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.828+0000 7f7f9ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:30:03.823 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.828+0000 7f7f9ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:03.823 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.828+0000 7f7f9ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:03.823 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.828+0000 7f7f9ffff640 10 monclient(hunting): my global_id is 4548 2024-03-29T11:30:03.823 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.828+0000 7f7f9ffff640 1 -- 172.21.15.114:0/2658068377 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fa002e320 con 0x7f7fa00c3070 2024-03-29T11:30:03.823 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.828+0000 7f7f9ffff640 1 -- 172.21.15.114:0/2658068377 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1533976665 0 0) 0x7f7f94003520 con 0x7f7fa00c5de0 2024-03-29T11:30:03.823 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.828+0000 7f7f9ffff640 1 -- 172.21.15.114:0/2658068377 >> v1:172.21.15.176:6789/0 conn(0x7f7fa0036a60 legacy=0x7f7fa00c56d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:03.823 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.828+0000 7f7f9ffff640 1 -- 172.21.15.114:0/2658068377 >> v1:172.21.15.114:6789/0 conn(0x7f7fa00c3070 legacy=0x7f7fa00d0fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:03.823 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.828+0000 7f7f9ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:30:03.823 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.828+0000 7f7f9ffff640 1 monclient: found mon.noname-b 2024-03-29T11:30:03.824 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.828+0000 7f7f9ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:03.824 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.828+0000 7f7f9ffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.114:6790/0 2024-03-29T11:30:03.824 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.828+0000 7f7f9ffff640 1 -- 172.21.15.114:0/2658068377 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fa002c830 con 0x7f7fa00c5de0 2024-03-29T11:30:03.824 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.828+0000 7f7f9ffff640 10 monclient: _finish_auth 0 2024-03-29T11:30:03.824 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.828+0000 7f7f9ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:30:03.824 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.828+0000 7f7f9ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:03.824 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.828+0000 7f7f9ffff640 1 -- 172.21.15.114:0/2658068377 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f7f98001690 con 0x7f7fa00c5de0 2024-03-29T11:30:03.824 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.828+0000 7f7f9ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:03.824 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.829+0000 7f7f9ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:30:03.824 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.829+0000 7f7f9ffff640 10 monclient: dump: 2024-03-29T11:30:03.824 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:30:03.825 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:30:03.825 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:30:03.825 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:30:03.825 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:03.825 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:30:03.825 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:30:03.825 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:30:03.825 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:30:03.825 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:30:03.825 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.829+0000 7f7f9ffff640 1 -- 172.21.15.114:0/2658068377 <== mon.2 v1:172.21.15.114:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7f980018e0 con 0x7f7fa00c5de0 2024-03-29T11:30:03.825 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.829+0000 7f7f9ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:03.825 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.829+0000 7f7f9ffff640 1 -- 172.21.15.114:0/2658068377 <== mon.2 v1:172.21.15.114:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f7f98001c70 con 0x7f7fa00c5de0 2024-03-29T11:30:03.825 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.829+0000 7f7f9ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:03.825 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.829+0000 7f7f9ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:30:03.826 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.829+0000 7f7f9ffff640 10 monclient: dump: 2024-03-29T11:30:03.826 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:30:03.826 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:30:03.826 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:30:03.826 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:30:03.826 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:03.826 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:30:03.826 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:30:03.826 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:30:03.826 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:30:03.826 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:30:03.826 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.829+0000 7f7fae169640 5 monclient: authenticate success, global_id 4580 2024-03-29T11:30:03.826 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.829+0000 7f7fae169640 10 monclient: get_monmap_and_config success 2024-03-29T11:30:03.826 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.829+0000 7f7fae169640 4 set_mon_vals no callback set 2024-03-29T11:30:03.826 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.830+0000 7f7fae169640 10 monclient: shutdown 2024-03-29T11:30:03.827 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.830+0000 7f7fae169640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:03.827 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.830+0000 7f7fae169640 1 -- 172.21.15.114:0/2658068377 >> v1:172.21.15.114:6790/0 conn(0x7f7fa00c5de0 legacy=0x7f7fa00c2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:03.827 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.830+0000 7f7fae169640 1 -- 172.21.15.114:0/2658068377 shutdown_connections 2024-03-29T11:30:03.827 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.830+0000 7f7fae169640 1 -- 172.21.15.114:0/2658068377 >> 172.21.15.114:0/2658068377 conn(0x7f7fa0005c00 msgr2=0x7f7fa0006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:03.828 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.832+0000 7f7fae169640 1 -- 172.21.15.114:0/2658068377 shutdown_connections 2024-03-29T11:30:03.828 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.835+0000 7f7fae169640 1 -- 172.21.15.114:0/2658068377 wait complete. 2024-03-29T11:30:03.831 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.836+0000 7f7fae169640 10 monclient: build_initial_monmap 2024-03-29T11:30:03.831 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.836+0000 7f7fae169640 10 monclient: monmap: 2024-03-29T11:30:03.831 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:30:03.831 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:30:03.831 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T11:30:03.831 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T11:30:03.831 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:03.831 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:30:03.832 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:30:03.832 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T11:30:03.832 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T11:30:03.832 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:30:03.832 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.836+0000 7f7fae169640 1 librados: starting msgr at 2024-03-29T11:30:03.832 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.836+0000 7f7fae169640 1 librados: starting objecter 2024-03-29T11:30:03.832 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.836+0000 7f7fae169640 1 Processor -- start 2024-03-29T11:30:03.832 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.837+0000 7f7fae169640 1 -- start start 2024-03-29T11:30:03.832 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.837+0000 7f7fae169640 1 librados: setting wanted keys 2024-03-29T11:30:03.832 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.837+0000 7f7fae169640 1 librados: calling monclient init 2024-03-29T11:30:03.832 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.837+0000 7f7fae169640 10 monclient: init 2024-03-29T11:30:03.832 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.837+0000 7f7fae169640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:03.832 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.837+0000 7f7fae169640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:30:03.832 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.837+0000 7f7fae169640 10 monclient(hunting): picked mon.noname-c con 0x7f7fa0036a60 addr v1:172.21.15.114:6790/0 2024-03-29T11:30:03.833 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.837+0000 7f7fae169640 10 monclient(hunting): picked mon.noname-a con 0x7f7fa00c3070 addr v1:172.21.15.114:6789/0 2024-03-29T11:30:03.833 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.837+0000 7f7fae169640 10 monclient(hunting): picked mon.noname-b con 0x7f7fa00bb960 addr v1:172.21.15.176:6789/0 2024-03-29T11:30:03.833 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.837+0000 7f7fae169640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fa00bc380 con 0x7f7fa00c3070 2024-03-29T11:30:03.833 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.837+0000 7f7fae169640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fa00c6a10 con 0x7f7fa00bb960 2024-03-29T11:30:03.833 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.837+0000 7f7fae169640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fa00c7fc0 con 0x7f7fa0036a60 2024-03-29T11:30:03.833 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.837+0000 7f7fae169640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:03.833 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.837+0000 7f7fae169640 10 monclient(hunting): authenticate will time out at 1474.020020s 2024-03-29T11:30:03.835 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.841+0000 7f7fad167640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f7fa0036a60 0x7f7fa00c89a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:51726/0 (socket says 172.21.15.114:51726) 2024-03-29T11:30:03.835 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.841+0000 7f7fad167640 1 -- 172.21.15.114:0/2379101583 learned_addr learned my addr 172.21.15.114:0/2379101583 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:30:03.836 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.841+0000 7f7f9dffb640 10 client.?.objecter ms_handle_connect 0x7f7fa0036a60 2024-03-29T11:30:03.836 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.841+0000 7f7f9dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:03.836 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.841+0000 7f7f9dffb640 10 client.?.objecter ms_handle_connect 0x7f7fa00c3070 2024-03-29T11:30:03.836 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.841+0000 7f7f9dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:03.836 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.841+0000 7f7f9dffb640 1 -- 172.21.15.114:0/2379101583 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4131569431 0 0) 0x7f7fa00c7fc0 con 0x7f7fa0036a60 2024-03-29T11:30:03.836 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.841+0000 7f7f9dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:30:03.836 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.841+0000 7f7f9dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:03.836 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.841+0000 7f7f9dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:03.836 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.841+0000 7f7f9dffb640 10 monclient(hunting): my global_id is 4583 2024-03-29T11:30:03.836 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.841+0000 7f7f9dffb640 1 -- 172.21.15.114:0/2379101583 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f8c004eb0 con 0x7f7fa0036a60 2024-03-29T11:30:03.836 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.841+0000 7f7f9dffb640 1 -- 172.21.15.114:0/2379101583 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2527710545 0 0) 0x7f7fa00bc380 con 0x7f7fa00c3070 2024-03-29T11:30:03.836 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.841+0000 7f7f9dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:30:03.836 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.841+0000 7f7f9dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:03.836 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.841+0000 7f7f9dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:03.837 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.841+0000 7f7f9dffb640 10 monclient(hunting): my global_id is 4551 2024-03-29T11:30:03.837 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.841+0000 7f7f9dffb640 1 -- 172.21.15.114:0/2379101583 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fa00c7fc0 con 0x7f7fa00c3070 2024-03-29T11:30:03.837 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.842+0000 7f7f9dffb640 1 -- 172.21.15.114:0/2379101583 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1111281481 0 0) 0x7f7f8c004eb0 con 0x7f7fa0036a60 2024-03-29T11:30:03.837 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.842+0000 7f7f9dffb640 1 -- 172.21.15.114:0/2379101583 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7fa00bc380 con 0x7f7fa0036a60 2024-03-29T11:30:03.837 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.842+0000 7f7f9dffb640 1 -- 172.21.15.114:0/2379101583 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f7fa804fdd0 con 0x7f7fa0036a60 2024-03-29T11:30:03.837 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.842+0000 7f7f9dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:03.837 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.842+0000 7f7f9dffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:30:03.837 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.842+0000 7f7f9dffb640 10 monclient(hunting): dump: 2024-03-29T11:30:03.837 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:30:03.837 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:30:03.837 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:30:03.837 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:30:03.837 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:03.837 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:30:03.838 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:30:03.838 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:30:03.838 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:30:03.838 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:30:03.838 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.842+0000 7f7f9dffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:30:03.838 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.842+0000 7f7f9dffb640 10 client.?.objecter ms_handle_connect 0x7f7fa00bb960 2024-03-29T11:30:03.838 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.842+0000 7f7f9dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:03.838 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.842+0000 7f7f9dffb640 1 -- 172.21.15.114:0/2379101583 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2658289660 0 0) 0x7f7fa00c7fc0 con 0x7f7fa00c3070 2024-03-29T11:30:03.838 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.842+0000 7f7f9dffb640 1 -- 172.21.15.114:0/2379101583 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7f8c004eb0 con 0x7f7fa00c3070 2024-03-29T11:30:03.838 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.842+0000 7f7f9dffb640 1 -- 172.21.15.114:0/2379101583 <== mon.2 v1:172.21.15.114:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 650353432 0 0) 0x7f7fa00bc380 con 0x7f7fa0036a60 2024-03-29T11:30:03.838 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.842+0000 7f7f9dffb640 1 -- 172.21.15.114:0/2379101583 >> v1:172.21.15.176:6789/0 conn(0x7f7fa00bb960 legacy=0x7f7fa016b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:03.839 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.842+0000 7f7f9dffb640 1 -- 172.21.15.114:0/2379101583 >> v1:172.21.15.114:6789/0 conn(0x7f7fa00c3070 legacy=0x7f7fa00c6ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:03.840 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.842+0000 7f7f9dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:30:03.840 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.842+0000 7f7f9dffb640 1 monclient: found mon.c 2024-03-29T11:30:03.840 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.842+0000 7f7f9dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:03.840 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.842+0000 7f7f9dffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:30:03.840 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.842+0000 7f7f9dffb640 1 -- 172.21.15.114:0/2379101583 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fa00c82a0 con 0x7f7fa0036a60 2024-03-29T11:30:03.840 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.842+0000 7f7f9dffb640 10 monclient: _finish_auth 0 2024-03-29T11:30:03.840 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.842+0000 7f7f9dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:30:03.840 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.842+0000 7f7f9dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:03.843 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.843+0000 7f7fae169640 5 monclient: authenticate success, global_id 4583 2024-03-29T11:30:03.844 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.843+0000 7f7fae169640 10 monclient: _renew_subs 2024-03-29T11:30:03.844 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.843+0000 7f7fae169640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:30:03.844 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.843+0000 7f7fae169640 1 -- 172.21.15.114:0/2379101583 --> v1:172.21.15.114:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fa00c8580 con 0x7f7fa0036a60 2024-03-29T11:30:03.844 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.843+0000 7f7fae169640 10 client.4583.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:30:03.844 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.843+0000 7f7fae169640 10 monclient: _renew_subs 2024-03-29T11:30:03.844 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.843+0000 7f7fae169640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:30:03.844 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.843+0000 7f7fae169640 1 -- 172.21.15.114:0/2379101583 --> v1:172.21.15.114:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fa016cfe0 con 0x7f7fa0036a60 2024-03-29T11:30:03.844 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.843+0000 7f7fae169640 1 librados: init done 2024-03-29T11:30:03.844 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.845+0000 7f7fae169640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:03.844 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.845+0000 7f7fae169640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:03.844 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.845+0000 7f7fae169640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:30:03.844 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.845+0000 7f7fae169640 1 -- 172.21.15.114:0/2379101583 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fa0005120 con 0x7f7fa0036a60 2024-03-29T11:30:03.844 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.846+0000 7f7f9dffb640 1 -- 172.21.15.114:0/2379101583 <== mon.2 v1:172.21.15.114:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7fa804f400 con 0x7f7fa0036a60 2024-03-29T11:30:03.844 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.846+0000 7f7f9dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:03.844 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.846+0000 7f7f9f7fe640 4 set_mon_vals no callback set 2024-03-29T11:30:03.845 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.847+0000 7f7f9dffb640 1 -- 172.21.15.114:0/2379101583 <== mon.2 v1:172.21.15.114:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f7fa8075710 con 0x7f7fa0036a60 2024-03-29T11:30:03.845 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.847+0000 7f7f9dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:03.845 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.847+0000 7f7f9dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:30:03.845 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.847+0000 7f7f9dffb640 10 monclient: dump: 2024-03-29T11:30:03.845 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:30:03.845 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:30:03.845 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:30:03.845 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:30:03.845 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:03.845 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:30:03.845 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:30:03.845 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:30:03.845 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:30:03.845 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:30:03.845 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.848+0000 7f7f9dffb640 1 -- 172.21.15.114:0/2379101583 <== mon.2 v1:172.21.15.114:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1594292461 0 0) 0x7f7fa8067bf0 con 0x7f7fa0036a60 2024-03-29T11:30:03.846 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.849+0000 7f7fac966640 10 monclient: get_auth_request con 0x7f7f8c06dc30 auth_method 0 2024-03-29T11:30:03.846 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.849+0000 7f7f9dffb640 10 client.4583.objecter ms_handle_connect 0x7f7f8c06dc30 2024-03-29T11:30:03.846 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.849+0000 7f7f9dffb640 1 -- 172.21.15.114:0/2379101583 <== mon.2 v1:172.21.15.114:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3238987313 0 0) 0x7f7fa80ff490 con 0x7f7fa0036a60 2024-03-29T11:30:03.846 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.849+0000 7f7f9dffb640 10 client.4583.objecter ms_dispatch 0x7f7fa8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:30:03.846 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.850+0000 7f7f9dffb640 3 client.4583.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:30:03.846 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.850+0000 7f7f9dffb640 3 client.4583.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:30:03.846 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.850+0000 7f7f9dffb640 20 client.4583.objecter dump_active .. 0 homeless 2024-03-29T11:30:03.847 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.855+0000 7f7f9dffb640 1 -- 172.21.15.114:0/2379101583 <== mon.2 v1:172.21.15.114:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7fa80cad70 con 0x7f7fa0036a60 2024-03-29T11:30:03.847 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.855+0000 7f7f9dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:03.847 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:03.855+0000 7f7f9dffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:30:04.168 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:04.175+0000 7f7fae169640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:04.168 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:04.175+0000 7f7fae169640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:04.168 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:04.175+0000 7f7fae169640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:30:04.168 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:04.175+0000 7f7fae169640 1 -- 172.21.15.114:0/2379101583 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7fa00bf500 con 0x7f7fa0036a60 2024-03-29T11:30:04.169 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:04.176+0000 7f7f9dffb640 1 -- 172.21.15.114:0/2379101583 <== mon.2 v1:172.21.15.114:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2858224613) 0x7f7fa80ccca0 con 0x7f7fa0036a60 2024-03-29T11:30:04.169 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:04.176+0000 7f7f9dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:04.169 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:04.176+0000 7f7f9dffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:30:04.169 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T11:30:04.170 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"b0de2b9b-c45a-44c7-b6ee-e83b39a1e108","created":"2024-03-29T11:20:49.420660+0000","modified":"2024-03-29T11:20:51.700167+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:51.700167+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a64785d3-6b11-499a-b706-cc827ab96a74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5882dadb-e7a8-4604-ba9a-418013c106c8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d7e8c409-bc66-4e4a-bbdb-420bad31388e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0682989-3595-4cd6-90a5-ab9111c278e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"67068120-e3ff-4817-bd06-44feb7479a35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09e8b-8a85-4997-bdea-75d754e6ee67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2f0d9ead-f47f-4cb3-92a4-0a5ad23cd831","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fdb6a25f-6bd8-4c65-9b9b-b460ba7673a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:30:04.172 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:04.180+0000 7f7fae169640 10 librados: watch_flush enter 2024-03-29T11:30:04.173 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:04.180+0000 7f7fae169640 10 librados: watch_flush exit 2024-03-29T11:30:04.173 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:04.180+0000 7f7fae169640 20 client.4583.objecter shutdown clearing up homeless session... 2024-03-29T11:30:04.173 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:04.180+0000 7f7fae169640 10 client.4583.objecter successfully canceled tick 2024-03-29T11:30:04.174 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:04.181+0000 7f7fae169640 1 -- 172.21.15.114:0/2379101583 >> v1:172.21.15.114:6816/4038755055 conn(0x7f7f8c06dc30 legacy=0x7f7f8c0700f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:04.174 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:04.181+0000 7f7fae169640 10 monclient: shutdown 2024-03-29T11:30:04.174 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:04.181+0000 7f7fae169640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:04.174 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:04.181+0000 7f7fae169640 1 -- 172.21.15.114:0/2379101583 >> v1:172.21.15.114:6790/0 conn(0x7f7fa0036a60 legacy=0x7f7fa00c89a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:04.174 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:04.181+0000 7f7fae169640 1 -- 172.21.15.114:0/2379101583 shutdown_connections 2024-03-29T11:30:04.174 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:04.181+0000 7f7fae169640 1 -- 172.21.15.114:0/2379101583 >> 172.21.15.114:0/2379101583 conn(0x7f7fa0005c00 msgr2=0x7f7fa00c2730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:04.175 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:04.182+0000 7f7fae169640 1 -- 172.21.15.114:0/2379101583 shutdown_connections 2024-03-29T11:30:04.175 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:04.182+0000 7f7fae169640 1 -- 172.21.15.114:0/2379101583 wait complete. 2024-03-29T11:30:04.175 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:04.182+0000 7f7fae169640 1 librados: shutdown 2024-03-29T11:30:04.191 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:30:10.997 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:30:11.144 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.151+0000 7fcf712e0640 10 monclient: get_monmap_and_config 2024-03-29T11:30:11.144 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.151+0000 7fcf712e0640 10 monclient: build_initial_monmap 2024-03-29T11:30:11.144 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.152+0000 7fcf712e0640 10 monclient: monmap: 2024-03-29T11:30:11.144 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:30:11.144 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:30:11.144 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:30:11.153045+0000 2024-03-29T11:30:11.145 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:30:11.153045+0000 2024-03-29T11:30:11.145 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:11.145 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:30:11.145 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:30:11.145 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T11:30:11.145 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T11:30:11.145 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:30:11.145 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.153+0000 7fcf712e0640 1 Processor -- start 2024-03-29T11:30:11.146 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.153+0000 7fcf712e0640 1 -- start start 2024-03-29T11:30:11.146 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.153+0000 7fcf712e0640 10 monclient: init 2024-03-29T11:30:11.150 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.154+0000 7fcf712e0640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:11.150 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.154+0000 7fcf712e0640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:30:11.150 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.154+0000 7fcf712e0640 10 monclient(hunting): picked mon.noname-b con 0x7fcf6c06c250 addr v1:172.21.15.114:6790/0 2024-03-29T11:30:11.150 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.154+0000 7fcf712e0640 10 monclient(hunting): picked mon.noname-a con 0x7fcf6c072400 addr v1:172.21.15.114:6789/0 2024-03-29T11:30:11.150 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.154+0000 7fcf712e0640 10 monclient(hunting): picked mon.noname-c con 0x7fcf6c06c630 addr v1:172.21.15.176:6789/0 2024-03-29T11:30:11.150 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.154+0000 7fcf712e0640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf6c06d760 con 0x7fcf6c072400 2024-03-29T11:30:11.150 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.154+0000 7fcf712e0640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf6c070500 con 0x7fcf6c06c630 2024-03-29T11:30:11.150 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.154+0000 7fcf712e0640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf6c0aff50 con 0x7fcf6c06c250 2024-03-29T11:30:11.151 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.154+0000 7fcf712e0640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:11.151 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.154+0000 7fcf712e0640 10 monclient(hunting): authenticate will time out at 1481.336618s 2024-03-29T11:30:11.151 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.155+0000 7fcf63fff640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fcf6c072400 0x7fcf6c1465b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:57644/0 (socket says 172.21.15.114:57644) 2024-03-29T11:30:11.151 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.155+0000 7fcf63fff640 1 -- 172.21.15.114:0/405672968 learned_addr learned my addr 172.21.15.114:0/405672968 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:30:11.151 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.155+0000 7fcf6a7fc640 1 -- 172.21.15.114:0/405672968 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2385322083 0 0) 0x7fcf6c0aff50 con 0x7fcf6c06c250 2024-03-29T11:30:11.151 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.155+0000 7fcf6a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:30:11.151 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.155+0000 7fcf6a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:11.151 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.155+0000 7fcf6a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:11.152 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.155+0000 7fcf6a7fc640 10 monclient(hunting): my global_id is 4586 2024-03-29T11:30:11.152 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.155+0000 7fcf6a7fc640 1 -- 172.21.15.114:0/405672968 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf4c003520 con 0x7fcf6c06c250 2024-03-29T11:30:11.152 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.155+0000 7fcf6a7fc640 1 -- 172.21.15.114:0/405672968 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1622880312 0 0) 0x7fcf6c06d760 con 0x7fcf6c072400 2024-03-29T11:30:11.152 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.155+0000 7fcf6a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:30:11.152 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.155+0000 7fcf6a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:11.152 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.155+0000 7fcf6a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:11.152 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.155+0000 7fcf6a7fc640 10 monclient(hunting): my global_id is 4554 2024-03-29T11:30:11.152 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.155+0000 7fcf6a7fc640 1 -- 172.21.15.114:0/405672968 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf6c0aff50 con 0x7fcf6c072400 2024-03-29T11:30:11.152 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.156+0000 7fcf6a7fc640 1 -- 172.21.15.114:0/405672968 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2290144310 0 0) 0x7fcf4c003520 con 0x7fcf6c06c250 2024-03-29T11:30:11.152 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.156+0000 7fcf6a7fc640 1 -- 172.21.15.114:0/405672968 >> v1:172.21.15.176:6789/0 conn(0x7fcf6c06c630 legacy=0x7fcf6c151ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:30:11.152 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.156+0000 7fcf6a7fc640 1 -- 172.21.15.114:0/405672968 >> v1:172.21.15.114:6789/0 conn(0x7fcf6c072400 legacy=0x7fcf6c1465b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:11.153 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.156+0000 7fcf6a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:30:11.153 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.156+0000 7fcf6a7fc640 1 monclient: found mon.noname-b 2024-03-29T11:30:11.153 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.156+0000 7fcf6a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:11.153 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.156+0000 7fcf6a7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.114:6790/0 2024-03-29T11:30:11.153 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.156+0000 7fcf6a7fc640 1 -- 172.21.15.114:0/405672968 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf6c0b00e0 con 0x7fcf6c06c250 2024-03-29T11:30:11.153 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.156+0000 7fcf6a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:30:11.153 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.156+0000 7fcf6a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:30:11.153 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.156+0000 7fcf6a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:11.153 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.156+0000 7fcf712e0640 5 monclient: authenticate success, global_id 4586 2024-03-29T11:30:11.153 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.156+0000 7fcf712e0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:30:11.153 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.156+0000 7fcf6a7fc640 1 -- 172.21.15.114:0/405672968 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fcf54001690 con 0x7fcf6c06c250 2024-03-29T11:30:11.153 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.156+0000 7fcf6a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:11.153 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.156+0000 7fcf6a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:30:11.153 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.156+0000 7fcf6a7fc640 10 monclient: dump: 2024-03-29T11:30:11.153 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:30:11.154 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:30:11.154 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:30:11.154 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:30:11.154 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:11.154 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:30:11.154 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:30:11.154 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:30:11.154 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:30:11.154 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:30:11.154 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.156+0000 7fcf712e0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:30:11.154 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.156+0000 7fcf6a7fc640 1 -- 172.21.15.114:0/405672968 <== mon.2 v1:172.21.15.114:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcf540018b0 con 0x7fcf6c06c250 2024-03-29T11:30:11.155 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.156+0000 7fcf6a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:11.155 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.156+0000 7fcf712e0640 10 monclient: get_monmap_and_config success 2024-03-29T11:30:11.155 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.156+0000 7fcf712e0640 4 set_mon_vals no callback set 2024-03-29T11:30:11.155 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.157+0000 7fcf712e0640 10 monclient: shutdown 2024-03-29T11:30:11.155 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.157+0000 7fcf712e0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:11.155 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.157+0000 7fcf712e0640 1 -- 172.21.15.114:0/405672968 >> v1:172.21.15.114:6790/0 conn(0x7fcf6c06c250 legacy=0x7fcf6c071cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:11.155 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.157+0000 7fcf712e0640 1 -- 172.21.15.114:0/405672968 shutdown_connections 2024-03-29T11:30:11.155 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.157+0000 7fcf712e0640 1 -- 172.21.15.114:0/405672968 >> 172.21.15.114:0/405672968 conn(0x7fcf6c067610 msgr2=0x7fcf6c0b5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:11.155 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.157+0000 7fcf712e0640 1 -- 172.21.15.114:0/405672968 shutdown_connections 2024-03-29T11:30:11.155 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.157+0000 7fcf712e0640 1 -- 172.21.15.114:0/405672968 wait complete. 2024-03-29T11:30:11.166 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.158+0000 7fcf712e0640 10 monclient: build_initial_monmap 2024-03-29T11:30:11.166 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.158+0000 7fcf712e0640 10 monclient: monmap: 2024-03-29T11:30:11.166 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:30:11.167 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:30:11.167 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T11:30:11.167 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T11:30:11.167 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:11.167 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:30:11.167 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:30:11.167 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T11:30:11.167 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T11:30:11.167 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:30:11.167 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.158+0000 7fcf712e0640 1 librados: starting msgr at 2024-03-29T11:30:11.167 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.158+0000 7fcf712e0640 1 librados: starting objecter 2024-03-29T11:30:11.168 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.159+0000 7fcf712e0640 1 Processor -- start 2024-03-29T11:30:11.168 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.159+0000 7fcf712e0640 1 -- start start 2024-03-29T11:30:11.168 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.159+0000 7fcf712e0640 1 librados: setting wanted keys 2024-03-29T11:30:11.168 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.159+0000 7fcf712e0640 1 librados: calling monclient init 2024-03-29T11:30:11.168 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.159+0000 7fcf712e0640 10 monclient: init 2024-03-29T11:30:11.168 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.159+0000 7fcf712e0640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:11.168 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.159+0000 7fcf712e0640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:30:11.168 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.160+0000 7fcf712e0640 10 monclient(hunting): picked mon.noname-a con 0x7fcf6c06c250 addr v1:172.21.15.114:6789/0 2024-03-29T11:30:11.168 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.160+0000 7fcf712e0640 10 monclient(hunting): picked mon.noname-c con 0x7fcf6c06c630 addr v1:172.21.15.114:6790/0 2024-03-29T11:30:11.168 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.160+0000 7fcf712e0640 10 monclient(hunting): picked mon.noname-b con 0x7fcf6c072400 addr v1:172.21.15.176:6789/0 2024-03-29T11:30:11.168 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.160+0000 7fcf712e0640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf6c0abc80 con 0x7fcf6c06c250 2024-03-29T11:30:11.168 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.160+0000 7fcf712e0640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf6c0aac20 con 0x7fcf6c072400 2024-03-29T11:30:11.168 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.160+0000 7fcf712e0640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf6c0b3610 con 0x7fcf6c06c630 2024-03-29T11:30:11.168 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.160+0000 7fcf712e0640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:11.169 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.160+0000 7fcf712e0640 10 monclient(hunting): authenticate will time out at 1481.342180s 2024-03-29T11:30:11.169 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.160+0000 7fcf63fff640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7fcf6c06c630 0x7fcf6c1dbbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:52034/0 (socket says 172.21.15.114:52034) 2024-03-29T11:30:11.169 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.160+0000 7fcf63fff640 1 -- 172.21.15.114:0/2390108275 learned_addr learned my addr 172.21.15.114:0/2390108275 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:30:11.169 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.160+0000 7fcf637fe640 10 client.?.objecter ms_handle_connect 0x7fcf6c072400 2024-03-29T11:30:11.169 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.160+0000 7fcf637fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:11.169 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.160+0000 7fcf637fe640 10 client.?.objecter ms_handle_connect 0x7fcf6c06c630 2024-03-29T11:30:11.169 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.160+0000 7fcf637fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:11.169 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.160+0000 7fcf637fe640 1 -- 172.21.15.114:0/2390108275 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1856699729 0 0) 0x7fcf6c0b3610 con 0x7fcf6c06c630 2024-03-29T11:30:11.169 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.160+0000 7fcf637fe640 10 monclient(hunting): _negotiate 2024-03-29T11:30:11.169 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.160+0000 7fcf637fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:11.169 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.160+0000 7fcf637fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:11.169 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.160+0000 7fcf637fe640 10 monclient(hunting): my global_id is 4589 2024-03-29T11:30:11.169 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.160+0000 7fcf637fe640 1 -- 172.21.15.114:0/2390108275 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf50004eb0 con 0x7fcf6c06c630 2024-03-29T11:30:11.169 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.160+0000 7fcf637fe640 1 -- 172.21.15.114:0/2390108275 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3816310541 0 0) 0x7fcf6c0aac20 con 0x7fcf6c072400 2024-03-29T11:30:11.170 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.160+0000 7fcf637fe640 10 monclient(hunting): _negotiate 2024-03-29T11:30:11.170 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.160+0000 7fcf637fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:11.170 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.160+0000 7fcf637fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:11.170 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.160+0000 7fcf637fe640 10 monclient(hunting): my global_id is 4531 2024-03-29T11:30:11.170 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.160+0000 7fcf637fe640 1 -- 172.21.15.114:0/2390108275 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf6c0b3610 con 0x7fcf6c072400 2024-03-29T11:30:11.170 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.161+0000 7fcf637fe640 10 client.?.objecter ms_handle_connect 0x7fcf6c06c250 2024-03-29T11:30:11.170 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.161+0000 7fcf637fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:11.170 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.161+0000 7fcf637fe640 1 -- 172.21.15.114:0/2390108275 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4248016326 0 0) 0x7fcf6c0abc80 con 0x7fcf6c06c250 2024-03-29T11:30:11.170 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.161+0000 7fcf637fe640 10 monclient(hunting): _negotiate 2024-03-29T11:30:11.170 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.161+0000 7fcf637fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:11.170 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.161+0000 7fcf637fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:11.171 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.161+0000 7fcf637fe640 10 monclient(hunting): my global_id is 4557 2024-03-29T11:30:11.171 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.161+0000 7fcf637fe640 1 -- 172.21.15.114:0/2390108275 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf6c0aac20 con 0x7fcf6c06c250 2024-03-29T11:30:11.171 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.161+0000 7fcf637fe640 1 -- 172.21.15.114:0/2390108275 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3945327188 0 0) 0x7fcf6c0b3610 con 0x7fcf6c072400 2024-03-29T11:30:11.171 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.161+0000 7fcf637fe640 1 -- 172.21.15.114:0/2390108275 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcf50006680 con 0x7fcf6c072400 2024-03-29T11:30:11.171 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.161+0000 7fcf637fe640 1 -- 172.21.15.114:0/2390108275 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fcf64001890 con 0x7fcf6c072400 2024-03-29T11:30:11.171 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.161+0000 7fcf637fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:11.171 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.161+0000 7fcf637fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:30:11.171 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.161+0000 7fcf637fe640 10 monclient(hunting): dump: 2024-03-29T11:30:11.171 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:30:11.171 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:30:11.171 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:30:11.171 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:30:11.171 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:11.171 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:30:11.172 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:30:11.172 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:30:11.172 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:30:11.172 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:30:11.172 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.161+0000 7fcf637fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:30:11.172 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.161+0000 7fcf637fe640 1 -- 172.21.15.114:0/2390108275 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3856540070 0 0) 0x7fcf50004eb0 con 0x7fcf6c06c630 2024-03-29T11:30:11.172 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.161+0000 7fcf637fe640 1 -- 172.21.15.114:0/2390108275 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcf6c0b3610 con 0x7fcf6c06c630 2024-03-29T11:30:11.172 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.161+0000 7fcf637fe640 1 -- 172.21.15.114:0/2390108275 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fcf5c001910 con 0x7fcf6c06c630 2024-03-29T11:30:11.172 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.161+0000 7fcf637fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:11.172 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.161+0000 7fcf637fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:30:11.172 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.161+0000 7fcf637fe640 10 monclient(hunting): dump: 2024-03-29T11:30:11.172 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:30:11.172 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:30:11.172 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:30:11.172 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:30:11.173 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:11.173 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:30:11.173 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:30:11.173 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:30:11.173 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:30:11.173 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:30:11.173 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.161+0000 7fcf637fe640 1 -- 172.21.15.114:0/2390108275 <== mon.1 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3328895969 0 0) 0x7fcf50006680 con 0x7fcf6c072400 2024-03-29T11:30:11.173 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.161+0000 7fcf637fe640 1 -- 172.21.15.114:0/2390108275 >> v1:172.21.15.114:6790/0 conn(0x7fcf6c06c630 legacy=0x7fcf6c1dbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:11.173 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.161+0000 7fcf637fe640 1 -- 172.21.15.114:0/2390108275 >> v1:172.21.15.114:6789/0 conn(0x7fcf6c06c250 legacy=0x7fcf6c0aa510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:11.173 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.161+0000 7fcf637fe640 10 monclient: _finish_hunting 0 2024-03-29T11:30:11.173 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.161+0000 7fcf637fe640 1 monclient: found mon.b 2024-03-29T11:30:11.173 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.161+0000 7fcf637fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:11.173 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.161+0000 7fcf637fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:30:11.173 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.161+0000 7fcf637fe640 1 -- 172.21.15.114:0/2390108275 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf6c0b37a0 con 0x7fcf6c072400 2024-03-29T11:30:11.173 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.161+0000 7fcf637fe640 10 monclient: _finish_auth 0 2024-03-29T11:30:11.174 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.161+0000 7fcf637fe640 10 monclient: _check_auth_tickets 2024-03-29T11:30:11.174 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.161+0000 7fcf637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:11.174 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.161+0000 7fcf712e0640 5 monclient: authenticate success, global_id 4531 2024-03-29T11:30:11.174 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.161+0000 7fcf712e0640 10 monclient: _renew_subs 2024-03-29T11:30:11.174 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.161+0000 7fcf712e0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:30:11.174 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.161+0000 7fcf712e0640 1 -- 172.21.15.114:0/2390108275 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf6c1dc2f0 con 0x7fcf6c072400 2024-03-29T11:30:11.174 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.161+0000 7fcf712e0640 10 client.4531.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:30:11.174 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.161+0000 7fcf712e0640 10 monclient: _renew_subs 2024-03-29T11:30:11.174 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.161+0000 7fcf712e0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:30:11.174 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.161+0000 7fcf712e0640 1 -- 172.21.15.114:0/2390108275 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf6c0a9a70 con 0x7fcf6c072400 2024-03-29T11:30:11.174 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.161+0000 7fcf712e0640 1 librados: init done 2024-03-29T11:30:11.174 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.162+0000 7fcf637fe640 1 -- 172.21.15.114:0/2390108275 <== mon.1 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcf640019f0 con 0x7fcf6c072400 2024-03-29T11:30:11.174 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.162+0000 7fcf637fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:11.174 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.162+0000 7fcf637fe640 1 -- 172.21.15.114:0/2390108275 <== mon.1 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fcf64002040 con 0x7fcf6c072400 2024-03-29T11:30:11.175 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.162+0000 7fcf637fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:11.175 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.162+0000 7fcf637fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:30:11.175 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.162+0000 7fcf637fe640 10 monclient: dump: 2024-03-29T11:30:11.175 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:30:11.175 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:30:11.175 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:30:11.175 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:30:11.175 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:11.175 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:30:11.175 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:30:11.175 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:30:11.175 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:30:11.175 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:30:11.175 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.162+0000 7fcf69ffb640 4 set_mon_vals no callback set 2024-03-29T11:30:11.176 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.163+0000 7fcf712e0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:11.176 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.163+0000 7fcf712e0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:11.176 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.163+0000 7fcf712e0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:30:11.176 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.163+0000 7fcf712e0640 1 -- 172.21.15.114:0/2390108275 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf44005080 con 0x7fcf6c072400 2024-03-29T11:30:11.176 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.164+0000 7fcf637fe640 1 -- 172.21.15.114:0/2390108275 <== mon.1 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1594292461 0 0) 0x7fcf64019a80 con 0x7fcf6c072400 2024-03-29T11:30:11.176 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.165+0000 7fcf637fe640 1 -- 172.21.15.114:0/2390108275 <== mon.1 v1:172.21.15.176:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3238987313 0 0) 0x7fcf64083a30 con 0x7fcf6c072400 2024-03-29T11:30:11.176 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.165+0000 7fcf637fe640 10 client.4531.objecter ms_dispatch 0x7fcf6c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:30:11.176 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.165+0000 7fcf637fe640 3 client.4531.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:30:11.176 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.165+0000 7fcf637fe640 3 client.4531.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:30:11.176 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.165+0000 7fcf637fe640 20 client.4531.objecter dump_active .. 0 homeless 2024-03-29T11:30:11.176 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.165+0000 7fcf6affd640 10 monclient: get_auth_request con 0x7fcf5006e100 auth_method 0 2024-03-29T11:30:11.176 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.166+0000 7fcf637fe640 10 client.4531.objecter ms_handle_connect 0x7fcf5006e100 2024-03-29T11:30:11.176 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.173+0000 7fcf637fe640 1 -- 172.21.15.114:0/2390108275 <== mon.1 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fcf6404f140 con 0x7fcf6c072400 2024-03-29T11:30:11.176 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.173+0000 7fcf637fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:11.176 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.173+0000 7fcf637fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:30:11.564 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.571+0000 7fcf712e0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:11.564 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.571+0000 7fcf712e0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:11.564 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.571+0000 7fcf712e0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:30:11.564 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.571+0000 7fcf712e0640 1 -- 172.21.15.114:0/2390108275 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcf44005330 con 0x7fcf6c072400 2024-03-29T11:30:11.566 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.573+0000 7fcf637fe640 1 -- 172.21.15.114:0/2390108275 <== mon.1 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2858224613) 0x7fcf64051240 con 0x7fcf6c072400 2024-03-29T11:30:11.566 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.573+0000 7fcf637fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:11.566 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.573+0000 7fcf637fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:30:11.567 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T11:30:11.567 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"b0de2b9b-c45a-44c7-b6ee-e83b39a1e108","created":"2024-03-29T11:20:49.420660+0000","modified":"2024-03-29T11:20:51.700167+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:51.700167+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a64785d3-6b11-499a-b706-cc827ab96a74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5882dadb-e7a8-4604-ba9a-418013c106c8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d7e8c409-bc66-4e4a-bbdb-420bad31388e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0682989-3595-4cd6-90a5-ab9111c278e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"67068120-e3ff-4817-bd06-44feb7479a35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09e8b-8a85-4997-bdea-75d754e6ee67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2f0d9ead-f47f-4cb3-92a4-0a5ad23cd831","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fdb6a25f-6bd8-4c65-9b9b-b460ba7673a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:30:11.570 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.578+0000 7fcf712e0640 10 librados: watch_flush enter 2024-03-29T11:30:11.571 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.578+0000 7fcf712e0640 10 librados: watch_flush exit 2024-03-29T11:30:11.571 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.578+0000 7fcf712e0640 20 client.4531.objecter shutdown clearing up homeless session... 2024-03-29T11:30:11.571 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.578+0000 7fcf712e0640 10 client.4531.objecter successfully canceled tick 2024-03-29T11:30:11.571 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.578+0000 7fcf712e0640 1 -- 172.21.15.114:0/2390108275 >> v1:172.21.15.114:6816/4038755055 conn(0x7fcf5006e100 legacy=0x7fcf500705c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:11.571 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.578+0000 7fcf712e0640 10 monclient: shutdown 2024-03-29T11:30:11.571 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.578+0000 7fcf712e0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:11.571 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.578+0000 7fcf712e0640 1 -- 172.21.15.114:0/2390108275 >> v1:172.21.15.176:6789/0 conn(0x7fcf6c072400 legacy=0x7fcf6c1e6160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:11.571 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.578+0000 7fcf712e0640 1 -- 172.21.15.114:0/2390108275 shutdown_connections 2024-03-29T11:30:11.571 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.578+0000 7fcf712e0640 1 -- 172.21.15.114:0/2390108275 >> 172.21.15.114:0/2390108275 conn(0x7fcf6c067610 msgr2=0x7fcf6c149ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:11.571 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.578+0000 7fcf712e0640 1 -- 172.21.15.114:0/2390108275 shutdown_connections 2024-03-29T11:30:11.572 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.579+0000 7fcf712e0640 1 -- 172.21.15.114:0/2390108275 wait complete. 2024-03-29T11:30:11.572 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:11.579+0000 7fcf712e0640 1 librados: shutdown 2024-03-29T11:30:11.586 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:30:18.394 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:30:18.553 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.557+0000 7fea31c4e640 10 monclient: get_monmap_and_config 2024-03-29T11:30:18.553 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.557+0000 7fea31c4e640 10 monclient: build_initial_monmap 2024-03-29T11:30:18.553 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.557+0000 7fea31c4e640 10 monclient: monmap: 2024-03-29T11:30:18.553 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:30:18.553 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:30:18.553 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:30:18.559003+0000 2024-03-29T11:30:18.554 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:30:18.559003+0000 2024-03-29T11:30:18.554 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:18.554 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:30:18.554 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:30:18.554 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T11:30:18.554 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T11:30:18.555 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:30:18.555 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.559+0000 7fea31c4e640 1 Processor -- start 2024-03-29T11:30:18.555 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.560+0000 7fea31c4e640 1 -- start start 2024-03-29T11:30:18.555 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.560+0000 7fea31c4e640 10 monclient: init 2024-03-29T11:30:18.556 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.561+0000 7fea31c4e640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:18.556 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.561+0000 7fea31c4e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:30:18.556 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.561+0000 7fea31c4e640 10 monclient(hunting): picked mon.noname-b con 0x7fea2c0b4100 addr v1:172.21.15.114:6790/0 2024-03-29T11:30:18.556 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.561+0000 7fea31c4e640 10 monclient(hunting): picked mon.noname-a con 0x7fea2c13ee90 addr v1:172.21.15.114:6789/0 2024-03-29T11:30:18.556 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.561+0000 7fea31c4e640 10 monclient(hunting): picked mon.noname-c con 0x7fea2c0a9ac0 addr v1:172.21.15.176:6789/0 2024-03-29T11:30:18.556 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.561+0000 7fea31c4e640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea2c0af6f0 con 0x7fea2c13ee90 2024-03-29T11:30:18.556 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.561+0000 7fea31c4e640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea2c0aac80 con 0x7fea2c0a9ac0 2024-03-29T11:30:18.556 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.561+0000 7fea31c4e640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea2c0aaf60 con 0x7fea2c0b4100 2024-03-29T11:30:18.557 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.561+0000 7fea31c4e640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:18.557 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.561+0000 7fea31c4e640 10 monclient(hunting): authenticate will time out at 1488.743483s 2024-03-29T11:30:18.559 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.566+0000 7fea2affd640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fea2c13ee90 0x7fea2c13b5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:43675/0 (socket says 172.21.15.114:43675) 2024-03-29T11:30:18.560 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.566+0000 7fea2affd640 1 -- 172.21.15.114:0/2235202356 learned_addr learned my addr 172.21.15.114:0/2235202356 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:30:18.560 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.566+0000 7fea2a7fc640 1 -- 172.21.15.114:0/2235202356 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4136111709 0 0) 0x7fea2c0af6f0 con 0x7fea2c13ee90 2024-03-29T11:30:18.560 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.566+0000 7fea2a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:30:18.560 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.566+0000 7fea2a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:18.561 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.566+0000 7fea2a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:18.562 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.566+0000 7fea2a7fc640 10 monclient(hunting): my global_id is 4560 2024-03-29T11:30:18.562 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.566+0000 7fea2a7fc640 1 -- 172.21.15.114:0/2235202356 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea14003520 con 0x7fea2c13ee90 2024-03-29T11:30:18.562 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.568+0000 7fea2a7fc640 1 -- 172.21.15.114:0/2235202356 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 783874375 0 0) 0x7fea14003520 con 0x7fea2c13ee90 2024-03-29T11:30:18.562 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.568+0000 7fea2a7fc640 1 -- 172.21.15.114:0/2235202356 >> v1:172.21.15.114:6790/0 conn(0x7fea2c0b4100 legacy=0x7fea2c0adfe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:30:18.564 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.569+0000 7fea2a7fc640 1 -- 172.21.15.114:0/2235202356 >> v1:172.21.15.176:6789/0 conn(0x7fea2c0a9ac0 legacy=0x7fea2c142ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:18.564 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.569+0000 7fea2a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:30:18.565 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.569+0000 7fea2a7fc640 1 monclient: found mon.noname-a 2024-03-29T11:30:18.565 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.569+0000 7fea2a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:18.565 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.569+0000 7fea2a7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.114:6789/0 2024-03-29T11:30:18.565 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.569+0000 7fea2a7fc640 1 -- 172.21.15.114:0/2235202356 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea2c13e970 con 0x7fea2c13ee90 2024-03-29T11:30:18.565 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.569+0000 7fea2a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:30:18.565 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.569+0000 7fea2a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:30:18.565 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.570+0000 7fea2a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:18.565 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.570+0000 7fea2a7fc640 1 -- 172.21.15.114:0/2235202356 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fea24001a10 con 0x7fea2c13ee90 2024-03-29T11:30:18.566 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.570+0000 7fea2a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:18.566 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.570+0000 7fea2a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:30:18.566 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.570+0000 7fea2a7fc640 10 monclient: dump: 2024-03-29T11:30:18.566 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:30:18.566 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:30:18.566 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:30:18.567 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:30:18.567 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:18.567 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:30:18.567 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:30:18.567 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:30:18.567 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:30:18.567 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:30:18.568 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.570+0000 7fea31c4e640 5 monclient: authenticate success, global_id 4560 2024-03-29T11:30:18.568 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.570+0000 7fea31c4e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:30:18.568 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.570+0000 7fea2a7fc640 1 -- 172.21.15.114:0/2235202356 <== mon.0 v1:172.21.15.114:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fea24001ca0 con 0x7fea2c13ee90 2024-03-29T11:30:18.568 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.570+0000 7fea2a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:18.568 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.570+0000 7fea31c4e640 10 monclient: get_monmap_and_config success 2024-03-29T11:30:18.568 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.570+0000 7fea31c4e640 4 set_mon_vals no callback set 2024-03-29T11:30:18.568 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.571+0000 7fea31c4e640 10 monclient: shutdown 2024-03-29T11:30:18.569 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.571+0000 7fea31c4e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:18.569 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.571+0000 7fea31c4e640 1 -- 172.21.15.114:0/2235202356 >> v1:172.21.15.114:6789/0 conn(0x7fea2c13ee90 legacy=0x7fea2c13b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:18.569 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.571+0000 7fea31c4e640 1 -- 172.21.15.114:0/2235202356 shutdown_connections 2024-03-29T11:30:18.569 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.571+0000 7fea31c4e640 1 -- 172.21.15.114:0/2235202356 >> 172.21.15.114:0/2235202356 conn(0x7fea2c0b35e0 msgr2=0x7fea2c0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:18.570 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.574+0000 7fea31c4e640 1 -- 172.21.15.114:0/2235202356 shutdown_connections 2024-03-29T11:30:18.570 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.574+0000 7fea31c4e640 1 -- 172.21.15.114:0/2235202356 wait complete. 2024-03-29T11:30:18.570 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.575+0000 7fea31c4e640 10 monclient: build_initial_monmap 2024-03-29T11:30:18.570 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.575+0000 7fea31c4e640 10 monclient: monmap: 2024-03-29T11:30:18.570 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:30:18.570 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:30:18.571 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T11:30:18.571 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T11:30:18.571 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:18.571 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:30:18.571 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:30:18.571 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T11:30:18.571 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T11:30:18.571 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:30:18.571 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.575+0000 7fea31c4e640 1 librados: starting msgr at 2024-03-29T11:30:18.571 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.575+0000 7fea31c4e640 1 librados: starting objecter 2024-03-29T11:30:18.571 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.576+0000 7fea31c4e640 1 Processor -- start 2024-03-29T11:30:18.575 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.576+0000 7fea31c4e640 1 -- start start 2024-03-29T11:30:18.575 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.576+0000 7fea31c4e640 1 librados: setting wanted keys 2024-03-29T11:30:18.575 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.576+0000 7fea31c4e640 1 librados: calling monclient init 2024-03-29T11:30:18.575 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.576+0000 7fea31c4e640 10 monclient: init 2024-03-29T11:30:18.575 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.577+0000 7fea31c4e640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:18.575 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.577+0000 7fea31c4e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:30:18.575 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.577+0000 7fea31c4e640 10 monclient(hunting): picked mon.noname-a con 0x7fea2c0a9ac0 addr v1:172.21.15.114:6789/0 2024-03-29T11:30:18.575 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.577+0000 7fea31c4e640 10 monclient(hunting): picked mon.noname-b con 0x7fea2c0b4100 addr v1:172.21.15.176:6789/0 2024-03-29T11:30:18.575 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.577+0000 7fea31c4e640 10 monclient(hunting): picked mon.noname-c con 0x7fea2c12e3d0 addr v1:172.21.15.114:6790/0 2024-03-29T11:30:18.575 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.577+0000 7fea31c4e640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea2c12d430 con 0x7fea2c0a9ac0 2024-03-29T11:30:18.575 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.577+0000 7fea31c4e640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea2c12e820 con 0x7fea2c0b4100 2024-03-29T11:30:18.576 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.577+0000 7fea31c4e640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea2c12f110 con 0x7fea2c12e3d0 2024-03-29T11:30:18.576 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.577+0000 7fea31c4e640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:18.576 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.577+0000 7fea31c4e640 10 monclient(hunting): authenticate will time out at 1488.759293s 2024-03-29T11:30:18.576 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.581+0000 7fea2affd640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7fea2c0b4100 0x7fea2c06fc10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.114:44803/0 (socket says 172.21.15.114:44803) 2024-03-29T11:30:18.576 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.581+0000 7fea2affd640 1 -- 172.21.15.114:0/3909331076 learned_addr learned my addr 172.21.15.114:0/3909331076 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:30:18.576 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.581+0000 7fea13fff640 10 client.?.objecter ms_handle_connect 0x7fea2c0b4100 2024-03-29T11:30:18.576 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.581+0000 7fea13fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:18.576 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.581+0000 7fea13fff640 10 client.?.objecter ms_handle_connect 0x7fea2c12e3d0 2024-03-29T11:30:18.576 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.581+0000 7fea13fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:18.579 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.583+0000 7fea13fff640 1 -- 172.21.15.114:0/3909331076 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4110418708 0 0) 0x7fea2c12e820 con 0x7fea2c0b4100 2024-03-29T11:30:18.579 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.583+0000 7fea13fff640 10 monclient(hunting): _negotiate 2024-03-29T11:30:18.579 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.583+0000 7fea13fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:18.579 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.583+0000 7fea13fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:18.580 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.583+0000 7fea13fff640 10 monclient(hunting): my global_id is 4537 2024-03-29T11:30:18.580 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.583+0000 7fea13fff640 1 -- 172.21.15.114:0/3909331076 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea18004eb0 con 0x7fea2c0b4100 2024-03-29T11:30:18.580 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.583+0000 7fea13fff640 1 -- 172.21.15.114:0/3909331076 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 206034019 0 0) 0x7fea2c12f110 con 0x7fea2c12e3d0 2024-03-29T11:30:18.580 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.583+0000 7fea13fff640 10 monclient(hunting): _negotiate 2024-03-29T11:30:18.580 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.583+0000 7fea13fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:18.580 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.583+0000 7fea13fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:18.581 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.583+0000 7fea13fff640 10 monclient(hunting): my global_id is 4592 2024-03-29T11:30:18.581 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.583+0000 7fea13fff640 1 -- 172.21.15.114:0/3909331076 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea2c12e820 con 0x7fea2c12e3d0 2024-03-29T11:30:18.581 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.584+0000 7fea13fff640 1 -- 172.21.15.114:0/3909331076 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3499680527 0 0) 0x7fea18004eb0 con 0x7fea2c0b4100 2024-03-29T11:30:18.581 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.584+0000 7fea13fff640 1 -- 172.21.15.114:0/3909331076 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fea2c12f110 con 0x7fea2c0b4100 2024-03-29T11:30:18.581 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.584+0000 7fea13fff640 1 -- 172.21.15.114:0/3909331076 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fea24001cb0 con 0x7fea2c0b4100 2024-03-29T11:30:18.581 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.584+0000 7fea13fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:18.581 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.584+0000 7fea13fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:30:18.581 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.584+0000 7fea13fff640 10 monclient(hunting): dump: 2024-03-29T11:30:18.581 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:30:18.581 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:30:18.581 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:30:18.581 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:30:18.581 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:18.581 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:30:18.581 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:30:18.582 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:30:18.582 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:30:18.582 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:30:18.582 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.584+0000 7fea13fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:30:18.582 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.584+0000 7fea13fff640 1 -- 172.21.15.114:0/3909331076 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4130577437 0 0) 0x7fea2c12e820 con 0x7fea2c12e3d0 2024-03-29T11:30:18.582 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.584+0000 7fea13fff640 1 -- 172.21.15.114:0/3909331076 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fea18004eb0 con 0x7fea2c12e3d0 2024-03-29T11:30:18.582 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.585+0000 7fea13fff640 1 -- 172.21.15.114:0/3909331076 <== mon.1 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3544689334 0 0) 0x7fea2c12f110 con 0x7fea2c0b4100 2024-03-29T11:30:18.582 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.585+0000 7fea13fff640 1 -- 172.21.15.114:0/3909331076 >> v1:172.21.15.114:6790/0 conn(0x7fea2c12e3d0 legacy=0x7fea2c1e1fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:18.582 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.585+0000 7fea13fff640 1 -- 172.21.15.114:0/3909331076 >> v1:172.21.15.114:6789/0 conn(0x7fea2c0a9ac0 legacy=0x7fea2c12d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:18.582 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.585+0000 7fea13fff640 10 monclient: _finish_hunting 0 2024-03-29T11:30:18.582 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.585+0000 7fea13fff640 1 monclient: found mon.b 2024-03-29T11:30:18.582 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.585+0000 7fea13fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:18.582 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.585+0000 7fea13fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:30:18.582 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.585+0000 7fea13fff640 1 -- 172.21.15.114:0/3909331076 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea2c1318f0 con 0x7fea2c0b4100 2024-03-29T11:30:18.583 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.585+0000 7fea13fff640 10 monclient: _finish_auth 0 2024-03-29T11:30:18.583 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.585+0000 7fea13fff640 10 monclient: _check_auth_tickets 2024-03-29T11:30:18.583 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.585+0000 7fea13fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:18.583 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.585+0000 7fea31c4e640 5 monclient: authenticate success, global_id 4537 2024-03-29T11:30:18.583 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.585+0000 7fea31c4e640 10 monclient: _renew_subs 2024-03-29T11:30:18.583 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.585+0000 7fea31c4e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:30:18.583 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.585+0000 7fea31c4e640 1 -- 172.21.15.114:0/3909331076 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea2c12eda0 con 0x7fea2c0b4100 2024-03-29T11:30:18.583 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.586+0000 7fea31c4e640 10 client.4537.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:30:18.583 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.586+0000 7fea31c4e640 10 monclient: _renew_subs 2024-03-29T11:30:18.583 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.586+0000 7fea31c4e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:30:18.583 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.586+0000 7fea31c4e640 1 -- 172.21.15.114:0/3909331076 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fea2c130520 con 0x7fea2c0b4100 2024-03-29T11:30:18.583 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.586+0000 7fea31c4e640 1 librados: init done 2024-03-29T11:30:18.585 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.588+0000 7fea31c4e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:18.585 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.588+0000 7fea31c4e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:18.585 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.588+0000 7fea31c4e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:30:18.585 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.588+0000 7fea31c4e640 1 -- 172.21.15.114:0/3909331076 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea2c1316f0 con 0x7fea2c0b4100 2024-03-29T11:30:18.585 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.589+0000 7fea13fff640 1 -- 172.21.15.114:0/3909331076 <== mon.1 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fea240050e0 con 0x7fea2c0b4100 2024-03-29T11:30:18.585 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.589+0000 7fea13fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:18.585 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.589+0000 7fea13fff640 1 -- 172.21.15.114:0/3909331076 <== mon.1 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fea24005470 con 0x7fea2c0b4100 2024-03-29T11:30:18.586 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.589+0000 7fea13fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:18.586 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.589+0000 7fea13fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:30:18.586 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.589+0000 7fea13fff640 10 monclient: dump: 2024-03-29T11:30:18.586 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:30:18.586 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:30:18.586 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:30:18.586 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:30:18.586 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:18.587 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:30:18.587 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:30:18.587 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:30:18.587 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:30:18.587 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:30:18.587 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.589+0000 7fea297fa640 4 set_mon_vals no callback set 2024-03-29T11:30:18.587 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.591+0000 7fea13fff640 1 -- 172.21.15.114:0/3909331076 <== mon.1 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1594292461 0 0) 0x7fea24001e90 con 0x7fea2c0b4100 2024-03-29T11:30:18.587 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.591+0000 7fea13fff640 1 -- 172.21.15.114:0/3909331076 <== mon.1 v1:172.21.15.176:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3238987313 0 0) 0x7fea24084e10 con 0x7fea2c0b4100 2024-03-29T11:30:18.587 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.591+0000 7fea13fff640 10 client.4537.objecter ms_dispatch 0x7fea2c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:30:18.587 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.591+0000 7fea13fff640 3 client.4537.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:30:18.587 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.591+0000 7fea13fff640 3 client.4537.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:30:18.587 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.591+0000 7fea13fff640 20 client.4537.objecter dump_active .. 0 homeless 2024-03-29T11:30:18.587 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.592+0000 7fea2b7fe640 10 monclient: get_auth_request con 0x7fea1806dec0 auth_method 0 2024-03-29T11:30:18.587 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.592+0000 7fea13fff640 10 client.4537.objecter ms_handle_connect 0x7fea1806dec0 2024-03-29T11:30:18.589 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.596+0000 7fea13fff640 1 -- 172.21.15.114:0/3909331076 <== mon.1 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fea240506f0 con 0x7fea2c0b4100 2024-03-29T11:30:18.589 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.596+0000 7fea13fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:18.589 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.596+0000 7fea13fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:30:18.945 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.953+0000 7fea31c4e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:18.945 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.953+0000 7fea31c4e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:18.946 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.953+0000 7fea31c4e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:30:18.946 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.953+0000 7fea31c4e640 1 -- 172.21.15.114:0/3909331076 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fea2c12ef30 con 0x7fea2c0b4100 2024-03-29T11:30:18.947 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.954+0000 7fea13fff640 1 -- 172.21.15.114:0/3909331076 <== mon.1 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2858224613) 0x7fea24052620 con 0x7fea2c0b4100 2024-03-29T11:30:18.947 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.954+0000 7fea13fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:18.947 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.954+0000 7fea13fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:30:18.947 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T11:30:18.947 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"b0de2b9b-c45a-44c7-b6ee-e83b39a1e108","created":"2024-03-29T11:20:49.420660+0000","modified":"2024-03-29T11:20:51.700167+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:51.700167+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a64785d3-6b11-499a-b706-cc827ab96a74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5882dadb-e7a8-4604-ba9a-418013c106c8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d7e8c409-bc66-4e4a-bbdb-420bad31388e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0682989-3595-4cd6-90a5-ab9111c278e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"67068120-e3ff-4817-bd06-44feb7479a35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09e8b-8a85-4997-bdea-75d754e6ee67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2f0d9ead-f47f-4cb3-92a4-0a5ad23cd831","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fdb6a25f-6bd8-4c65-9b9b-b460ba7673a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:30:18.951 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.958+0000 7fea31c4e640 10 librados: watch_flush enter 2024-03-29T11:30:18.951 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.958+0000 7fea31c4e640 10 librados: watch_flush exit 2024-03-29T11:30:18.951 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.958+0000 7fea31c4e640 20 client.4537.objecter shutdown clearing up homeless session... 2024-03-29T11:30:18.951 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.958+0000 7fea31c4e640 10 client.4537.objecter successfully canceled tick 2024-03-29T11:30:18.951 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.958+0000 7fea31c4e640 1 -- 172.21.15.114:0/3909331076 >> v1:172.21.15.114:6816/4038755055 conn(0x7fea1806dec0 legacy=0x7fea18070380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:18.951 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.958+0000 7fea31c4e640 10 monclient: shutdown 2024-03-29T11:30:18.952 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.958+0000 7fea31c4e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:18.952 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.958+0000 7fea31c4e640 1 -- 172.21.15.114:0/3909331076 >> v1:172.21.15.176:6789/0 conn(0x7fea2c0b4100 legacy=0x7fea2c06fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:18.952 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.958+0000 7fea31c4e640 1 -- 172.21.15.114:0/3909331076 shutdown_connections 2024-03-29T11:30:18.952 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.958+0000 7fea31c4e640 1 -- 172.21.15.114:0/3909331076 >> 172.21.15.114:0/3909331076 conn(0x7fea2c0b35e0 msgr2=0x7fea2c0b07c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:18.952 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.959+0000 7fea31c4e640 1 -- 172.21.15.114:0/3909331076 shutdown_connections 2024-03-29T11:30:18.952 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.959+0000 7fea31c4e640 1 -- 172.21.15.114:0/3909331076 wait complete. 2024-03-29T11:30:18.952 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:18.959+0000 7fea31c4e640 1 librados: shutdown 2024-03-29T11:30:18.966 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:30:25.773 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:30:25.927 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.934+0000 7fefdc9dc640 10 monclient: get_monmap_and_config 2024-03-29T11:30:25.927 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.934+0000 7fefdc9dc640 10 monclient: build_initial_monmap 2024-03-29T11:30:25.928 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.934+0000 7fefdc9dc640 10 monclient: monmap: 2024-03-29T11:30:25.928 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:30:25.928 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:30:25.928 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:30:25.935932+0000 2024-03-29T11:30:25.928 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:30:25.935932+0000 2024-03-29T11:30:25.928 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:25.928 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:30:25.928 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:30:25.928 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T11:30:25.928 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T11:30:25.928 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:30:25.929 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.936+0000 7fefdc9dc640 1 Processor -- start 2024-03-29T11:30:25.929 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.936+0000 7fefdc9dc640 1 -- start start 2024-03-29T11:30:25.929 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.936+0000 7fefdc9dc640 10 monclient: init 2024-03-29T11:30:25.929 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.936+0000 7fefdc9dc640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:25.931 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.936+0000 7fefdc9dc640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:30:25.932 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.936+0000 7fefdc9dc640 10 monclient(hunting): picked mon.noname-b con 0x7fefd0006a70 addr v1:172.21.15.114:6790/0 2024-03-29T11:30:25.932 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.936+0000 7fefdc9dc640 10 monclient(hunting): picked mon.noname-a con 0x7fefd002b100 addr v1:172.21.15.114:6789/0 2024-03-29T11:30:25.932 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.937+0000 7fefdc9dc640 10 monclient(hunting): picked mon.noname-c con 0x7fefd0029c30 addr v1:172.21.15.176:6789/0 2024-03-29T11:30:25.932 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.937+0000 7fefdc9dc640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefd002a270 con 0x7fefd002b100 2024-03-29T11:30:25.932 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.937+0000 7fefdc9dc640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefd002c440 con 0x7fefd0029c30 2024-03-29T11:30:25.932 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.937+0000 7fefdc9dc640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefd00c9440 con 0x7fefd0006a70 2024-03-29T11:30:25.932 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.937+0000 7fefdc9dc640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:25.932 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.937+0000 7fefdc9dc640 10 monclient(hunting): authenticate will time out at 1496.119534s 2024-03-29T11:30:25.933 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.939+0000 7fefd77fe640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7fefd0006a70 0x7fefd00c8d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:36426/0 (socket says 172.21.15.114:36426) 2024-03-29T11:30:25.933 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.939+0000 7fefd77fe640 1 -- 172.21.15.114:0/3379133027 learned_addr learned my addr 172.21.15.114:0/3379133027 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:30:25.937 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.941+0000 7fefd67fc640 1 -- 172.21.15.114:0/3379133027 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 678912724 0 0) 0x7fefd002c440 con 0x7fefd0029c30 2024-03-29T11:30:25.937 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.941+0000 7fefd67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:30:25.937 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.941+0000 7fefd67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:25.937 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.941+0000 7fefd67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:25.937 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.941+0000 7fefd67fc640 10 monclient(hunting): my global_id is 4540 2024-03-29T11:30:25.937 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.941+0000 7fefd67fc640 1 -- 172.21.15.114:0/3379133027 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefc8003520 con 0x7fefd0029c30 2024-03-29T11:30:25.937 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.942+0000 7fefd67fc640 1 -- 172.21.15.114:0/3379133027 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3619680880 0 0) 0x7fefc8003520 con 0x7fefd0029c30 2024-03-29T11:30:25.937 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.942+0000 7fefd67fc640 1 -- 172.21.15.114:0/3379133027 >> v1:172.21.15.114:6790/0 conn(0x7fefd0006a70 legacy=0x7fefd00c8d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:30:25.938 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.942+0000 7fefd67fc640 1 -- 172.21.15.114:0/3379133027 >> v1:172.21.15.114:6789/0 conn(0x7fefd002b100 legacy=0x7fefd00cf100 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:30:25.938 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.942+0000 7fefd67fc640 10 monclient: _finish_hunting 0 2024-03-29T11:30:25.938 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.942+0000 7fefd67fc640 1 monclient: found mon.noname-c 2024-03-29T11:30:25.938 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.942+0000 7fefd67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:25.938 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.942+0000 7fefd67fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.176:6789/0 2024-03-29T11:30:25.938 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.942+0000 7fefd67fc640 1 -- 172.21.15.114:0/3379133027 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefd00c95d0 con 0x7fefd0029c30 2024-03-29T11:30:25.938 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.942+0000 7fefd67fc640 10 monclient: _finish_auth 0 2024-03-29T11:30:25.938 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.942+0000 7fefd67fc640 10 monclient: _check_auth_tickets 2024-03-29T11:30:25.938 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.942+0000 7fefd67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:25.938 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.942+0000 7fefd67fc640 1 -- 172.21.15.114:0/3379133027 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fefd804f730 con 0x7fefd0029c30 2024-03-29T11:30:25.938 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.942+0000 7fefd67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:25.939 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.942+0000 7fefd67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:30:25.939 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.942+0000 7fefd67fc640 10 monclient: dump: 2024-03-29T11:30:25.939 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:30:25.939 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:30:25.939 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:30:25.939 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:30:25.939 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:25.939 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:30:25.939 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:30:25.939 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:30:25.939 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:30:25.939 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:30:25.939 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.942+0000 7fefd67fc640 1 -- 172.21.15.114:0/3379133027 <== mon.1 v1:172.21.15.176:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fefd80008d0 con 0x7fefd0029c30 2024-03-29T11:30:25.939 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.942+0000 7fefd67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:25.940 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.942+0000 7fefd67fc640 1 -- 172.21.15.114:0/3379133027 <== mon.1 v1:172.21.15.176:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fefd80758b0 con 0x7fefd0029c30 2024-03-29T11:30:25.940 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.942+0000 7fefd67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:25.940 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.942+0000 7fefd67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:30:25.940 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.942+0000 7fefd67fc640 10 monclient: dump: 2024-03-29T11:30:25.940 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:30:25.940 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:30:25.940 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:30:25.940 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:30:25.940 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:25.940 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:30:25.940 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:30:25.940 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:30:25.940 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:30:25.940 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:30:25.941 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.943+0000 7fefdc9dc640 5 monclient: authenticate success, global_id 4540 2024-03-29T11:30:25.941 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.943+0000 7fefdc9dc640 10 monclient: get_monmap_and_config success 2024-03-29T11:30:25.941 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.943+0000 7fefdc9dc640 4 set_mon_vals no callback set 2024-03-29T11:30:25.941 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.944+0000 7fefdc9dc640 10 monclient: shutdown 2024-03-29T11:30:25.941 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.944+0000 7fefdc9dc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:25.941 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.944+0000 7fefdc9dc640 1 -- 172.21.15.114:0/3379133027 >> v1:172.21.15.176:6789/0 conn(0x7fefd0029c30 legacy=0x7fefd00d74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:25.941 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.944+0000 7fefdc9dc640 1 -- 172.21.15.114:0/3379133027 shutdown_connections 2024-03-29T11:30:25.941 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.944+0000 7fefdc9dc640 1 -- 172.21.15.114:0/3379133027 >> 172.21.15.114:0/3379133027 conn(0x7fefd00058f0 msgr2=0x7fefd0005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:25.941 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.944+0000 7fefdc9dc640 1 -- 172.21.15.114:0/3379133027 shutdown_connections 2024-03-29T11:30:25.943 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.945+0000 7fefdc9dc640 1 -- 172.21.15.114:0/3379133027 wait complete. 2024-03-29T11:30:25.943 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.946+0000 7fefdc9dc640 10 monclient: build_initial_monmap 2024-03-29T11:30:25.943 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.946+0000 7fefdc9dc640 10 monclient: monmap: 2024-03-29T11:30:25.943 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:30:25.944 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:30:25.944 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T11:30:25.944 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T11:30:25.944 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:25.944 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:30:25.944 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:30:25.944 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T11:30:25.944 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T11:30:25.944 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:30:25.944 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.946+0000 7fefdc9dc640 1 librados: starting msgr at 2024-03-29T11:30:25.944 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.946+0000 7fefdc9dc640 1 librados: starting objecter 2024-03-29T11:30:25.944 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.946+0000 7fefdc9dc640 1 Processor -- start 2024-03-29T11:30:25.944 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.947+0000 7fefdc9dc640 1 -- start start 2024-03-29T11:30:25.944 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.947+0000 7fefdc9dc640 1 librados: setting wanted keys 2024-03-29T11:30:25.945 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.947+0000 7fefdc9dc640 1 librados: calling monclient init 2024-03-29T11:30:25.945 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.947+0000 7fefdc9dc640 10 monclient: init 2024-03-29T11:30:25.945 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.947+0000 7fefdc9dc640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:25.945 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.947+0000 7fefdc9dc640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:30:25.945 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.947+0000 7fefdc9dc640 10 monclient(hunting): picked mon.noname-c con 0x7fefd0006a70 addr v1:172.21.15.114:6790/0 2024-03-29T11:30:25.945 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.948+0000 7fefdc9dc640 10 monclient(hunting): picked mon.noname-b con 0x7fefd0029c30 addr v1:172.21.15.176:6789/0 2024-03-29T11:30:25.945 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.948+0000 7fefdc9dc640 10 monclient(hunting): picked mon.noname-a con 0x7fefd0169890 addr v1:172.21.15.114:6789/0 2024-03-29T11:30:25.945 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.948+0000 7fefdc9dc640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefd002e840 con 0x7fefd0169890 2024-03-29T11:30:25.945 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.948+0000 7fefdc9dc640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefd002cd10 con 0x7fefd0029c30 2024-03-29T11:30:25.945 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.948+0000 7fefdc9dc640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefd002eab0 con 0x7fefd0006a70 2024-03-29T11:30:25.945 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.948+0000 7fefdc9dc640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:25.945 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.948+0000 7fefdc9dc640 10 monclient(hunting): authenticate will time out at 1496.130519s 2024-03-29T11:30:25.949 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.951+0000 7fefd7fff640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fefd0169890 0x7fefd0173be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:48403/0 (socket says 172.21.15.114:48403) 2024-03-29T11:30:25.949 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.951+0000 7fefd7fff640 1 -- 172.21.15.114:0/2876333042 learned_addr learned my addr 172.21.15.114:0/2876333042 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:30:25.949 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.951+0000 7fefc3fff640 10 client.?.objecter ms_handle_connect 0x7fefd0006a70 2024-03-29T11:30:25.949 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.951+0000 7fefc3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:25.949 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.951+0000 7fefc3fff640 10 client.?.objecter ms_handle_connect 0x7fefd0169890 2024-03-29T11:30:25.949 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.951+0000 7fefc3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:25.949 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.951+0000 7fefc3fff640 1 -- 172.21.15.114:0/2876333042 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3110152529 0 0) 0x7fefd002eab0 con 0x7fefd0006a70 2024-03-29T11:30:25.949 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.951+0000 7fefc3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:30:25.950 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.951+0000 7fefc3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:25.950 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.951+0000 7fefc3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:25.950 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.951+0000 7fefc3fff640 10 monclient(hunting): my global_id is 4595 2024-03-29T11:30:25.950 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.951+0000 7fefc3fff640 1 -- 172.21.15.114:0/2876333042 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefbc004eb0 con 0x7fefd0006a70 2024-03-29T11:30:25.950 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.952+0000 7fefc3fff640 1 -- 172.21.15.114:0/2876333042 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2037683738 0 0) 0x7fefbc004eb0 con 0x7fefd0006a70 2024-03-29T11:30:25.950 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.952+0000 7fefc3fff640 1 -- 172.21.15.114:0/2876333042 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fefbc005300 con 0x7fefd0006a70 2024-03-29T11:30:25.950 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.952+0000 7fefc3fff640 1 -- 172.21.15.114:0/2876333042 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fefcc001510 con 0x7fefd0006a70 2024-03-29T11:30:25.950 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.952+0000 7fefc3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:25.950 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.952+0000 7fefc3fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:30:25.950 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.952+0000 7fefc3fff640 10 monclient(hunting): dump: 2024-03-29T11:30:25.950 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:30:25.950 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:30:25.951 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:30:25.951 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:30:25.951 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:25.951 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:30:25.951 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:30:25.951 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:30:25.951 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:30:25.951 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:30:25.951 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.952+0000 7fefc3fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:30:25.951 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.952+0000 7fefc3fff640 1 -- 172.21.15.114:0/2876333042 <== mon.2 v1:172.21.15.114:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3200284023 0 0) 0x7fefbc005300 con 0x7fefd0006a70 2024-03-29T11:30:25.951 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.952+0000 7fefc3fff640 1 -- 172.21.15.114:0/2876333042 >> v1:172.21.15.176:6789/0 conn(0x7fefd0029c30 legacy=0x7fefd0169180 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:30:25.951 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.953+0000 7fefc3fff640 1 -- 172.21.15.114:0/2876333042 >> v1:172.21.15.114:6789/0 conn(0x7fefd0169890 legacy=0x7fefd0173be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:25.951 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.953+0000 7fefc3fff640 10 monclient: _finish_hunting 0 2024-03-29T11:30:25.951 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.953+0000 7fefc3fff640 1 monclient: found mon.c 2024-03-29T11:30:25.952 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.953+0000 7fefc3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:25.952 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.954+0000 7fefc3fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:30:25.952 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.954+0000 7fefc3fff640 1 -- 172.21.15.114:0/2876333042 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefd00cc980 con 0x7fefd0006a70 2024-03-29T11:30:25.952 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.954+0000 7fefc3fff640 10 monclient: _finish_auth 0 2024-03-29T11:30:25.952 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.954+0000 7fefc3fff640 10 monclient: _check_auth_tickets 2024-03-29T11:30:25.952 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.954+0000 7fefc3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:25.952 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.954+0000 7fefdc9dc640 5 monclient: authenticate success, global_id 4595 2024-03-29T11:30:25.952 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.954+0000 7fefdc9dc640 10 monclient: _renew_subs 2024-03-29T11:30:25.952 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.954+0000 7fefdc9dc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:30:25.952 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.954+0000 7fefdc9dc640 1 -- 172.21.15.114:0/2876333042 --> v1:172.21.15.114:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefd002d8e0 con 0x7fefd0006a70 2024-03-29T11:30:25.952 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.954+0000 7fefdc9dc640 10 client.4595.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:30:25.952 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.954+0000 7fefdc9dc640 10 monclient: _renew_subs 2024-03-29T11:30:25.952 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.954+0000 7fefdc9dc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:30:25.952 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.954+0000 7fefdc9dc640 1 -- 172.21.15.114:0/2876333042 --> v1:172.21.15.114:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fefd01754e0 con 0x7fefd0006a70 2024-03-29T11:30:25.953 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.954+0000 7fefdc9dc640 1 librados: init done 2024-03-29T11:30:25.953 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.954+0000 7fefc3fff640 1 -- 172.21.15.114:0/2876333042 <== mon.2 v1:172.21.15.114:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fefcc001b00 con 0x7fefd0006a70 2024-03-29T11:30:25.953 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.954+0000 7fefc3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:25.953 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.954+0000 7fefc3fff640 1 -- 172.21.15.114:0/2876333042 <== mon.2 v1:172.21.15.114:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fefcc001510 con 0x7fefd0006a70 2024-03-29T11:30:25.953 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.954+0000 7fefc3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:25.953 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.954+0000 7fefc3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:30:25.953 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.954+0000 7fefc3fff640 10 monclient: dump: 2024-03-29T11:30:25.953 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:30:25.953 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:30:25.953 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:30:25.953 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:30:25.953 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:25.953 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:30:25.953 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:30:25.953 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:30:25.954 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:30:25.954 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:30:25.954 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.955+0000 7fefc1ffb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:25.954 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.955+0000 7fefc1ffb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:25.954 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.955+0000 7fefc1ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:30:25.954 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.955+0000 7fefc1ffb640 1 -- 172.21.15.114:0/2876333042 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefb8005080 con 0x7fefd0006a70 2024-03-29T11:30:25.954 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.956+0000 7fefd5ffb640 4 set_mon_vals no callback set 2024-03-29T11:30:25.955 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.957+0000 7fefc3fff640 1 -- 172.21.15.114:0/2876333042 <== mon.2 v1:172.21.15.114:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1594292461 0 0) 0x7fefcc01a980 con 0x7fefd0006a70 2024-03-29T11:30:25.955 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.958+0000 7fefc3fff640 1 -- 172.21.15.114:0/2876333042 <== mon.2 v1:172.21.15.114:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3238987313 0 0) 0x7fefcc0846d0 con 0x7fefd0006a70 2024-03-29T11:30:25.955 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.958+0000 7fefc3fff640 10 client.4595.objecter ms_dispatch 0x7fefd8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:30:25.955 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.958+0000 7fefc3fff640 3 client.4595.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:30:25.955 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.958+0000 7fefc3fff640 3 client.4595.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:30:25.955 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.958+0000 7fefd6ffd640 10 monclient: get_auth_request con 0x7fefbc06e100 auth_method 0 2024-03-29T11:30:25.955 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.958+0000 7fefc3fff640 20 client.4595.objecter dump_active .. 0 homeless 2024-03-29T11:30:25.955 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.958+0000 7fefc3fff640 10 client.4595.objecter ms_handle_connect 0x7fefbc06e100 2024-03-29T11:30:25.958 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.966+0000 7fefc3fff640 1 -- 172.21.15.114:0/2876333042 <== mon.2 v1:172.21.15.114:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fefcc050010 con 0x7fefd0006a70 2024-03-29T11:30:25.959 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.966+0000 7fefc3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:25.959 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:25.966+0000 7fefc3fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:30:26.289 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:26.296+0000 7fefc1ffb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:26.289 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:26.296+0000 7fefc1ffb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:26.289 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:26.296+0000 7fefc1ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:30:26.289 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:26.296+0000 7fefc1ffb640 1 -- 172.21.15.114:0/2876333042 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fefb8005330 con 0x7fefd0006a70 2024-03-29T11:30:26.290 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:26.297+0000 7fefc3fff640 1 -- 172.21.15.114:0/2876333042 <== mon.2 v1:172.21.15.114:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2858224613) 0x7fefcc083d80 con 0x7fefd0006a70 2024-03-29T11:30:26.290 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:26.297+0000 7fefc3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:26.290 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:26.297+0000 7fefc3fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:30:26.291 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T11:30:26.291 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"b0de2b9b-c45a-44c7-b6ee-e83b39a1e108","created":"2024-03-29T11:20:49.420660+0000","modified":"2024-03-29T11:20:51.700167+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:51.700167+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a64785d3-6b11-499a-b706-cc827ab96a74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5882dadb-e7a8-4604-ba9a-418013c106c8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d7e8c409-bc66-4e4a-bbdb-420bad31388e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0682989-3595-4cd6-90a5-ab9111c278e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"67068120-e3ff-4817-bd06-44feb7479a35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09e8b-8a85-4997-bdea-75d754e6ee67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2f0d9ead-f47f-4cb3-92a4-0a5ad23cd831","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fdb6a25f-6bd8-4c65-9b9b-b460ba7673a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:30:26.293 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:26.301+0000 7fefc1ffb640 10 librados: watch_flush enter 2024-03-29T11:30:26.294 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:26.301+0000 7fefc1ffb640 10 librados: watch_flush exit 2024-03-29T11:30:26.294 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:26.301+0000 7fefc1ffb640 20 client.4595.objecter shutdown clearing up homeless session... 2024-03-29T11:30:26.294 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:26.301+0000 7fefc1ffb640 10 client.4595.objecter successfully canceled tick 2024-03-29T11:30:26.294 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:26.301+0000 7fefc1ffb640 1 -- 172.21.15.114:0/2876333042 >> v1:172.21.15.114:6816/4038755055 conn(0x7fefbc06e100 legacy=0x7fefbc0705c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:26.294 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:26.301+0000 7fefc1ffb640 10 monclient: shutdown 2024-03-29T11:30:26.294 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:26.301+0000 7fefc1ffb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:26.294 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:26.301+0000 7fefc1ffb640 1 -- 172.21.15.114:0/2876333042 >> v1:172.21.15.114:6790/0 conn(0x7fefd0006a70 legacy=0x7fefd002d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:26.295 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:26.302+0000 7fefc1ffb640 1 -- 172.21.15.114:0/2876333042 shutdown_connections 2024-03-29T11:30:26.295 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:26.302+0000 7fefc1ffb640 1 -- 172.21.15.114:0/2876333042 >> 172.21.15.114:0/2876333042 conn(0x7fefd00058f0 msgr2=0x7fefd00c3e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:26.295 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:26.302+0000 7fefc1ffb640 1 -- 172.21.15.114:0/2876333042 shutdown_connections 2024-03-29T11:30:26.295 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:26.302+0000 7fefc1ffb640 1 -- 172.21.15.114:0/2876333042 wait complete. 2024-03-29T11:30:26.295 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:26.303+0000 7fefc1ffb640 1 librados: shutdown 2024-03-29T11:30:26.311 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:30:33.119 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:30:33.286 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.292+0000 7fdd73fff640 10 monclient: get_monmap_and_config 2024-03-29T11:30:33.286 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.292+0000 7fdd73fff640 10 monclient: build_initial_monmap 2024-03-29T11:30:33.287 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.292+0000 7fdd73fff640 10 monclient: monmap: 2024-03-29T11:30:33.287 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:30:33.287 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:30:33.287 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:30:33.293705+0000 2024-03-29T11:30:33.287 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:30:33.293705+0000 2024-03-29T11:30:33.287 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:33.287 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:30:33.287 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:30:33.287 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T11:30:33.287 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T11:30:33.287 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:30:33.292 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.294+0000 7fdd73fff640 1 Processor -- start 2024-03-29T11:30:33.292 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.294+0000 7fdd73fff640 1 -- start start 2024-03-29T11:30:33.292 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.294+0000 7fdd73fff640 10 monclient: init 2024-03-29T11:30:33.292 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.294+0000 7fdd73fff640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:33.292 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.294+0000 7fdd73fff640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:30:33.292 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.294+0000 7fdd73fff640 10 monclient(hunting): picked mon.noname-b con 0x7fdd6c034850 addr v1:172.21.15.114:6790/0 2024-03-29T11:30:33.293 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.294+0000 7fdd73fff640 10 monclient(hunting): picked mon.noname-a con 0x7fdd6c0c9550 addr v1:172.21.15.114:6789/0 2024-03-29T11:30:33.293 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.294+0000 7fdd73fff640 10 monclient(hunting): picked mon.noname-c con 0x7fdd6c02aac0 addr v1:172.21.15.176:6789/0 2024-03-29T11:30:33.293 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.294+0000 7fdd73fff640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd6c02c440 con 0x7fdd6c0c9550 2024-03-29T11:30:33.293 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.294+0000 7fdd73fff640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd6c02aed0 con 0x7fdd6c02aac0 2024-03-29T11:30:33.293 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.294+0000 7fdd73fff640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd6c02b190 con 0x7fdd6c034850 2024-03-29T11:30:33.293 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.295+0000 7fdd73fff640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:33.293 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.295+0000 7fdd73fff640 10 monclient(hunting): authenticate will time out at 1503.477105s 2024-03-29T11:30:33.293 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.298+0000 7fdd737fe640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7fdd6c02aac0 0x7fdd6c0cc5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.114:54683/0 (socket says 172.21.15.114:54683) 2024-03-29T11:30:33.293 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.298+0000 7fdd737fe640 1 -- 172.21.15.114:0/1985437009 learned_addr learned my addr 172.21.15.114:0/1985437009 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:30:33.293 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.299+0000 7fdd71ffb640 1 -- 172.21.15.114:0/1985437009 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3909474538 0 0) 0x7fdd6c02aed0 con 0x7fdd6c02aac0 2024-03-29T11:30:33.293 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.299+0000 7fdd71ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:30:33.293 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.299+0000 7fdd71ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:33.294 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.299+0000 7fdd71ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:33.294 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.299+0000 7fdd71ffb640 10 monclient(hunting): my global_id is 4543 2024-03-29T11:30:33.294 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.299+0000 7fdd71ffb640 1 -- 172.21.15.114:0/1985437009 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd5c003520 con 0x7fdd6c02aac0 2024-03-29T11:30:33.296 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.301+0000 7fdd71ffb640 1 -- 172.21.15.114:0/1985437009 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3038027083 0 0) 0x7fdd5c003520 con 0x7fdd6c02aac0 2024-03-29T11:30:33.296 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.301+0000 7fdd71ffb640 1 -- 172.21.15.114:0/1985437009 >> v1:172.21.15.114:6790/0 conn(0x7fdd6c034850 legacy=0x7fdd6c0c8e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:30:33.297 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.301+0000 7fdd71ffb640 1 -- 172.21.15.114:0/1985437009 >> v1:172.21.15.114:6789/0 conn(0x7fdd6c0c9550 legacy=0x7fdd6c0cbea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:33.297 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.301+0000 7fdd71ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:30:33.297 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.301+0000 7fdd71ffb640 1 monclient: found mon.noname-c 2024-03-29T11:30:33.297 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.301+0000 7fdd71ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:33.297 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.301+0000 7fdd71ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.176:6789/0 2024-03-29T11:30:33.297 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.301+0000 7fdd71ffb640 1 -- 172.21.15.114:0/1985437009 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd6c033060 con 0x7fdd6c02aac0 2024-03-29T11:30:33.297 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.301+0000 7fdd71ffb640 10 monclient: _finish_auth 0 2024-03-29T11:30:33.297 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.301+0000 7fdd71ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:30:33.297 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.302+0000 7fdd71ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:33.297 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.302+0000 7fdd71ffb640 1 -- 172.21.15.114:0/1985437009 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fdd7404f730 con 0x7fdd6c02aac0 2024-03-29T11:30:33.298 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.302+0000 7fdd71ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:33.298 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.302+0000 7fdd71ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:30:33.298 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.302+0000 7fdd71ffb640 10 monclient: dump: 2024-03-29T11:30:33.298 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:30:33.298 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:30:33.298 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:30:33.298 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:30:33.298 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:33.298 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:30:33.298 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:30:33.298 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:30:33.299 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:30:33.299 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:30:33.299 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.302+0000 7fdd71ffb640 1 -- 172.21.15.114:0/1985437009 <== mon.1 v1:172.21.15.176:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdd740008d0 con 0x7fdd6c02aac0 2024-03-29T11:30:33.299 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.303+0000 7fdd71ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:33.299 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.303+0000 7fdd71ffb640 1 -- 172.21.15.114:0/1985437009 <== mon.1 v1:172.21.15.176:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fdd740758b0 con 0x7fdd6c02aac0 2024-03-29T11:30:33.299 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.303+0000 7fdd71ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:33.299 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.303+0000 7fdd71ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:30:33.299 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.303+0000 7fdd71ffb640 10 monclient: dump: 2024-03-29T11:30:33.299 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:30:33.299 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:30:33.299 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:30:33.299 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:30:33.300 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:33.300 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:30:33.300 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:30:33.300 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:30:33.300 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:30:33.300 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:30:33.301 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.303+0000 7fdd73fff640 5 monclient: authenticate success, global_id 4543 2024-03-29T11:30:33.301 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.303+0000 7fdd73fff640 10 monclient: get_monmap_and_config success 2024-03-29T11:30:33.301 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.303+0000 7fdd73fff640 4 set_mon_vals no callback set 2024-03-29T11:30:33.301 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.304+0000 7fdd73fff640 10 monclient: shutdown 2024-03-29T11:30:33.301 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.304+0000 7fdd73fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:33.301 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.304+0000 7fdd73fff640 1 -- 172.21.15.114:0/1985437009 >> v1:172.21.15.176:6789/0 conn(0x7fdd6c02aac0 legacy=0x7fdd6c0cc5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:33.302 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.305+0000 7fdd73fff640 1 -- 172.21.15.114:0/1985437009 shutdown_connections 2024-03-29T11:30:33.302 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.305+0000 7fdd73fff640 1 -- 172.21.15.114:0/1985437009 >> 172.21.15.114:0/1985437009 conn(0x7fdd6c0058f0 msgr2=0x7fdd6c005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:33.303 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.309+0000 7fdd73fff640 1 -- 172.21.15.114:0/1985437009 shutdown_connections 2024-03-29T11:30:33.303 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.311+0000 7fdd73fff640 1 -- 172.21.15.114:0/1985437009 wait complete. 2024-03-29T11:30:33.305 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.312+0000 7fdd73fff640 10 monclient: build_initial_monmap 2024-03-29T11:30:33.305 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.312+0000 7fdd73fff640 10 monclient: monmap: 2024-03-29T11:30:33.305 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:30:33.305 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:30:33.306 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T11:30:33.306 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T11:30:33.306 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:33.306 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:30:33.306 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:30:33.306 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T11:30:33.306 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T11:30:33.306 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:30:33.306 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.312+0000 7fdd73fff640 1 librados: starting msgr at 2024-03-29T11:30:33.306 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.312+0000 7fdd73fff640 1 librados: starting objecter 2024-03-29T11:30:33.306 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.312+0000 7fdd73fff640 1 Processor -- start 2024-03-29T11:30:33.307 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.313+0000 7fdd73fff640 1 -- start start 2024-03-29T11:30:33.307 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.313+0000 7fdd73fff640 1 librados: setting wanted keys 2024-03-29T11:30:33.307 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.313+0000 7fdd73fff640 1 librados: calling monclient init 2024-03-29T11:30:33.307 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.313+0000 7fdd73fff640 10 monclient: init 2024-03-29T11:30:33.307 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.313+0000 7fdd73fff640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:33.307 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.313+0000 7fdd73fff640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:30:33.307 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.313+0000 7fdd73fff640 10 monclient(hunting): picked mon.noname-b con 0x7fdd6c02aac0 addr v1:172.21.15.176:6789/0 2024-03-29T11:30:33.307 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.313+0000 7fdd73fff640 10 monclient(hunting): picked mon.noname-a con 0x7fdd6c034850 addr v1:172.21.15.114:6789/0 2024-03-29T11:30:33.307 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.313+0000 7fdd73fff640 10 monclient(hunting): picked mon.noname-c con 0x7fdd6c0c9550 addr v1:172.21.15.114:6790/0 2024-03-29T11:30:33.308 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.313+0000 7fdd73fff640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd6c0c3560 con 0x7fdd6c034850 2024-03-29T11:30:33.308 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.313+0000 7fdd73fff640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd6c160e60 con 0x7fdd6c02aac0 2024-03-29T11:30:33.308 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.313+0000 7fdd73fff640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd6c02e0a0 con 0x7fdd6c0c9550 2024-03-29T11:30:33.308 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.313+0000 7fdd73fff640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:33.308 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.313+0000 7fdd73fff640 10 monclient(hunting): authenticate will time out at 1503.495952s 2024-03-29T11:30:33.311 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.317+0000 7fdd72ffd640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7fdd6c02aac0 0x7fdd6c161380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.114:54839/0 (socket says 172.21.15.114:54839) 2024-03-29T11:30:33.311 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.317+0000 7fdd72ffd640 1 -- 172.21.15.114:0/3083205327 learned_addr learned my addr 172.21.15.114:0/3083205327 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:30:33.312 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.317+0000 7fdd5b7fe640 10 client.?.objecter ms_handle_connect 0x7fdd6c02aac0 2024-03-29T11:30:33.312 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.317+0000 7fdd5b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:33.312 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.317+0000 7fdd5b7fe640 1 -- 172.21.15.114:0/3083205327 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3153455383 0 0) 0x7fdd6c160e60 con 0x7fdd6c02aac0 2024-03-29T11:30:33.312 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.317+0000 7fdd5b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:30:33.312 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.317+0000 7fdd5b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:33.312 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.317+0000 7fdd5b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:33.312 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.317+0000 7fdd5b7fe640 10 monclient(hunting): my global_id is 4546 2024-03-29T11:30:33.312 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.317+0000 7fdd5b7fe640 1 -- 172.21.15.114:0/3083205327 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd60004eb0 con 0x7fdd6c02aac0 2024-03-29T11:30:33.312 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.318+0000 7fdd5b7fe640 10 client.?.objecter ms_handle_connect 0x7fdd6c0c9550 2024-03-29T11:30:33.312 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.318+0000 7fdd5b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:33.313 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.319+0000 7fdd5b7fe640 1 -- 172.21.15.114:0/3083205327 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4168078530 0 0) 0x7fdd60004eb0 con 0x7fdd6c02aac0 2024-03-29T11:30:33.313 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.319+0000 7fdd5b7fe640 1 -- 172.21.15.114:0/3083205327 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdd6c160e60 con 0x7fdd6c02aac0 2024-03-29T11:30:33.313 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.319+0000 7fdd5b7fe640 1 -- 172.21.15.114:0/3083205327 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fdd68001960 con 0x7fdd6c02aac0 2024-03-29T11:30:33.313 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.319+0000 7fdd5b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:33.313 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.319+0000 7fdd5b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:30:33.313 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.319+0000 7fdd5b7fe640 10 monclient(hunting): dump: 2024-03-29T11:30:33.313 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:30:33.313 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:30:33.313 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:30:33.313 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:30:33.313 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:33.313 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:30:33.313 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:30:33.313 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:30:33.314 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:30:33.314 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:30:33.314 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.319+0000 7fdd5b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:30:33.314 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.319+0000 7fdd5b7fe640 1 -- 172.21.15.114:0/3083205327 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 884618780 0 0) 0x7fdd6c02e0a0 con 0x7fdd6c0c9550 2024-03-29T11:30:33.314 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.319+0000 7fdd5b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:30:33.314 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.319+0000 7fdd5b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:33.314 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.319+0000 7fdd5b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:33.314 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.319+0000 7fdd5b7fe640 10 monclient(hunting): my global_id is 4598 2024-03-29T11:30:33.314 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.319+0000 7fdd5b7fe640 1 -- 172.21.15.114:0/3083205327 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd60004eb0 con 0x7fdd6c0c9550 2024-03-29T11:30:33.314 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.319+0000 7fdd5b7fe640 1 -- 172.21.15.114:0/3083205327 <== mon.1 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3575001641 0 0) 0x7fdd6c160e60 con 0x7fdd6c02aac0 2024-03-29T11:30:33.314 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.319+0000 7fdd5b7fe640 1 -- 172.21.15.114:0/3083205327 >> v1:172.21.15.114:6790/0 conn(0x7fdd6c0c9550 legacy=0x7fdd6c16c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:33.314 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.319+0000 7fdd5b7fe640 1 -- 172.21.15.114:0/3083205327 >> v1:172.21.15.114:6789/0 conn(0x7fdd6c034850 legacy=0x7fdd6c15de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:33.314 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.319+0000 7fdd5b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:30:33.314 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.319+0000 7fdd5b7fe640 1 monclient: found mon.b 2024-03-29T11:30:33.314 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.320+0000 7fdd5b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:33.315 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.320+0000 7fdd5b7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:30:33.315 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.320+0000 7fdd5b7fe640 1 -- 172.21.15.114:0/3083205327 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd6c161a90 con 0x7fdd6c02aac0 2024-03-29T11:30:33.315 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.320+0000 7fdd5b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:30:33.315 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.320+0000 7fdd5b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:30:33.315 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.320+0000 7fdd5b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:33.317 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.321+0000 7fdd5b7fe640 1 -- 172.21.15.114:0/3083205327 <== mon.1 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdd68001c80 con 0x7fdd6c02aac0 2024-03-29T11:30:33.317 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.321+0000 7fdd5b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:33.317 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.321+0000 7fdd5b7fe640 1 -- 172.21.15.114:0/3083205327 <== mon.1 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fdd68003a40 con 0x7fdd6c02aac0 2024-03-29T11:30:33.317 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.321+0000 7fdd5b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:33.317 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.321+0000 7fdd5b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:30:33.317 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.321+0000 7fdd5b7fe640 10 monclient: dump: 2024-03-29T11:30:33.317 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:30:33.317 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:30:33.317 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:30:33.317 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:30:33.317 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:33.317 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:30:33.317 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:30:33.317 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:30:33.318 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:30:33.318 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:30:33.318 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.321+0000 7fdd70ff9640 4 set_mon_vals no callback set 2024-03-29T11:30:33.318 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.321+0000 7fdd73fff640 5 monclient: authenticate success, global_id 4546 2024-03-29T11:30:33.318 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.321+0000 7fdd73fff640 10 monclient: _renew_subs 2024-03-29T11:30:33.318 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.321+0000 7fdd73fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:30:33.318 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.321+0000 7fdd73fff640 1 -- 172.21.15.114:0/3083205327 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd6c0c47c0 con 0x7fdd6c02aac0 2024-03-29T11:30:33.318 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.321+0000 7fdd73fff640 10 client.4546.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:30:33.318 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.321+0000 7fdd73fff640 10 monclient: _renew_subs 2024-03-29T11:30:33.318 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.321+0000 7fdd73fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:30:33.318 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.321+0000 7fdd73fff640 1 -- 172.21.15.114:0/3083205327 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd6c15d930 con 0x7fdd6c02aac0 2024-03-29T11:30:33.318 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.321+0000 7fdd73fff640 1 librados: init done 2024-03-29T11:30:33.318 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.323+0000 7fdd73fff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:33.318 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.323+0000 7fdd73fff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:33.319 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.323+0000 7fdd73fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:30:33.319 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.323+0000 7fdd73fff640 1 -- 172.21.15.114:0/3083205327 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd6c160cc0 con 0x7fdd6c02aac0 2024-03-29T11:30:33.319 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.324+0000 7fdd5b7fe640 1 -- 172.21.15.114:0/3083205327 <== mon.1 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1594292461 0 0) 0x7fdd68003ba0 con 0x7fdd6c02aac0 2024-03-29T11:30:33.320 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.325+0000 7fdd5b7fe640 1 -- 172.21.15.114:0/3083205327 <== mon.1 v1:172.21.15.176:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3238987313 0 0) 0x7fdd68084af0 con 0x7fdd6c02aac0 2024-03-29T11:30:33.320 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.325+0000 7fdd727fc640 10 monclient: get_auth_request con 0x7fdd6006db80 auth_method 0 2024-03-29T11:30:33.320 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.326+0000 7fdd5b7fe640 10 client.4546.objecter ms_dispatch 0x7fdd74000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:30:33.320 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.326+0000 7fdd5b7fe640 3 client.4546.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:30:33.320 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.326+0000 7fdd5b7fe640 3 client.4546.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:30:33.320 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.326+0000 7fdd5b7fe640 20 client.4546.objecter dump_active .. 0 homeless 2024-03-29T11:30:33.320 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.326+0000 7fdd5b7fe640 10 client.4546.objecter ms_handle_connect 0x7fdd6006db80 2024-03-29T11:30:33.326 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.333+0000 7fdd5b7fe640 1 -- 172.21.15.114:0/3083205327 <== mon.1 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdd6c160cc0 con 0x7fdd6c02aac0 2024-03-29T11:30:33.326 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.333+0000 7fdd5b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:33.326 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.333+0000 7fdd5b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:30:33.656 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.663+0000 7fdd73fff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:33.656 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.663+0000 7fdd73fff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:33.656 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.663+0000 7fdd73fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:30:33.656 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.663+0000 7fdd73fff640 1 -- 172.21.15.114:0/3083205327 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdd6c1610a0 con 0x7fdd6c02aac0 2024-03-29T11:30:33.657 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.664+0000 7fdd5b7fe640 1 -- 172.21.15.114:0/3083205327 <== mon.1 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2858224613) 0x7fdd6c1610a0 con 0x7fdd6c02aac0 2024-03-29T11:30:33.657 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.664+0000 7fdd5b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:33.657 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.664+0000 7fdd5b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:30:33.657 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T11:30:33.658 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"b0de2b9b-c45a-44c7-b6ee-e83b39a1e108","created":"2024-03-29T11:20:49.420660+0000","modified":"2024-03-29T11:20:51.700167+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:51.700167+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a64785d3-6b11-499a-b706-cc827ab96a74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5882dadb-e7a8-4604-ba9a-418013c106c8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d7e8c409-bc66-4e4a-bbdb-420bad31388e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0682989-3595-4cd6-90a5-ab9111c278e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"67068120-e3ff-4817-bd06-44feb7479a35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09e8b-8a85-4997-bdea-75d754e6ee67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2f0d9ead-f47f-4cb3-92a4-0a5ad23cd831","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fdb6a25f-6bd8-4c65-9b9b-b460ba7673a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:30:33.661 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.667+0000 7fdd73fff640 10 librados: watch_flush enter 2024-03-29T11:30:33.661 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.667+0000 7fdd73fff640 10 librados: watch_flush exit 2024-03-29T11:30:33.661 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.667+0000 7fdd73fff640 20 client.4546.objecter shutdown clearing up homeless session... 2024-03-29T11:30:33.661 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.667+0000 7fdd73fff640 10 client.4546.objecter successfully canceled tick 2024-03-29T11:30:33.661 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.668+0000 7fdd73fff640 1 -- 172.21.15.114:0/3083205327 >> v1:172.21.15.114:6816/4038755055 conn(0x7fdd6006db80 legacy=0x7fdd60070040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:33.661 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.668+0000 7fdd73fff640 10 monclient: shutdown 2024-03-29T11:30:33.661 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.668+0000 7fdd73fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:33.662 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.668+0000 7fdd73fff640 1 -- 172.21.15.114:0/3083205327 >> v1:172.21.15.176:6789/0 conn(0x7fdd6c02aac0 legacy=0x7fdd6c161380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:33.662 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.668+0000 7fdd73fff640 1 -- 172.21.15.114:0/3083205327 shutdown_connections 2024-03-29T11:30:33.662 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.668+0000 7fdd73fff640 1 -- 172.21.15.114:0/3083205327 >> 172.21.15.114:0/3083205327 conn(0x7fdd6c0058f0 msgr2=0x7fdd6c0043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:33.662 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.668+0000 7fdd73fff640 1 -- 172.21.15.114:0/3083205327 shutdown_connections 2024-03-29T11:30:33.662 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.668+0000 7fdd73fff640 1 -- 172.21.15.114:0/3083205327 wait complete. 2024-03-29T11:30:33.662 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:33.668+0000 7fdd73fff640 1 librados: shutdown 2024-03-29T11:30:33.678 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:30:40.487 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:30:40.633 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.640+0000 7f02d8cd9640 10 monclient: get_monmap_and_config 2024-03-29T11:30:40.633 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.640+0000 7f02d8cd9640 10 monclient: build_initial_monmap 2024-03-29T11:30:40.633 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.640+0000 7f02d8cd9640 10 monclient: monmap: 2024-03-29T11:30:40.633 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:30:40.633 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:30:40.633 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:30:40.641327+0000 2024-03-29T11:30:40.634 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:30:40.641327+0000 2024-03-29T11:30:40.634 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:40.634 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:30:40.634 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:30:40.634 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T11:30:40.634 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T11:30:40.634 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:30:40.635 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.641+0000 7f02d8cd9640 1 Processor -- start 2024-03-29T11:30:40.635 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.641+0000 7f02d8cd9640 1 -- start start 2024-03-29T11:30:40.635 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.641+0000 7f02d8cd9640 10 monclient: init 2024-03-29T11:30:40.636 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.642+0000 7f02d8cd9640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:40.636 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.642+0000 7f02d8cd9640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:30:40.636 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.642+0000 7f02d8cd9640 10 monclient(hunting): picked mon.noname-c con 0x7f02d41457f0 addr v1:172.21.15.176:6789/0 2024-03-29T11:30:40.637 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.642+0000 7f02d8cd9640 10 monclient(hunting): picked mon.noname-a con 0x7f02d413f650 addr v1:172.21.15.114:6789/0 2024-03-29T11:30:40.637 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.642+0000 7f02d8cd9640 10 monclient(hunting): picked mon.noname-b con 0x7f02d40a9d70 addr v1:172.21.15.114:6790/0 2024-03-29T11:30:40.637 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.642+0000 7f02d8cd9640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02d40aaef0 con 0x7f02d413f650 2024-03-29T11:30:40.637 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.642+0000 7f02d8cd9640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02d4145bd0 con 0x7f02d41457f0 2024-03-29T11:30:40.637 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.642+0000 7f02d8cd9640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02d4145eb0 con 0x7f02d40a9d70 2024-03-29T11:30:40.637 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.642+0000 7f02d8cd9640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:40.637 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.642+0000 7f02d8cd9640 10 monclient(hunting): authenticate will time out at 1510.824839s 2024-03-29T11:30:40.637 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.642+0000 7f02d259c640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f02d41457f0 0x7f02d413ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.114:40848/0 (socket says 172.21.15.114:40848) 2024-03-29T11:30:40.637 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.642+0000 7f02d259c640 1 -- 172.21.15.114:0/2857193764 learned_addr learned my addr 172.21.15.114:0/2857193764 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:30:40.637 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.643+0000 7f02d159a640 1 -- 172.21.15.114:0/2857193764 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 214884255 0 0) 0x7f02d4145bd0 con 0x7f02d41457f0 2024-03-29T11:30:40.637 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.643+0000 7f02d159a640 10 monclient(hunting): _negotiate 2024-03-29T11:30:40.637 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.643+0000 7f02d159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:40.637 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.643+0000 7f02d159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:40.637 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.643+0000 7f02d159a640 10 monclient(hunting): my global_id is 4549 2024-03-29T11:30:40.638 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.643+0000 7f02d159a640 1 -- 172.21.15.114:0/2857193764 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02bc003520 con 0x7f02d41457f0 2024-03-29T11:30:40.641 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.644+0000 7f02d159a640 1 -- 172.21.15.114:0/2857193764 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2698519719 0 0) 0x7f02d40aaef0 con 0x7f02d413f650 2024-03-29T11:30:40.641 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.644+0000 7f02d159a640 10 monclient(hunting): _negotiate 2024-03-29T11:30:40.641 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.644+0000 7f02d159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:40.641 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.644+0000 7f02d159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:40.641 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.644+0000 7f02d159a640 10 monclient(hunting): my global_id is 4566 2024-03-29T11:30:40.641 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.644+0000 7f02d159a640 1 -- 172.21.15.114:0/2857193764 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02d4145bd0 con 0x7f02d413f650 2024-03-29T11:30:40.641 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.644+0000 7f02d159a640 1 -- 172.21.15.114:0/2857193764 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 87309494 0 0) 0x7f02d4145eb0 con 0x7f02d40a9d70 2024-03-29T11:30:40.642 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.644+0000 7f02d159a640 10 monclient(hunting): _negotiate 2024-03-29T11:30:40.642 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.644+0000 7f02d159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:40.642 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.644+0000 7f02d159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:40.642 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.644+0000 7f02d159a640 10 monclient(hunting): my global_id is 4601 2024-03-29T11:30:40.642 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.644+0000 7f02d159a640 1 -- 172.21.15.114:0/2857193764 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02d40aaef0 con 0x7f02d40a9d70 2024-03-29T11:30:40.642 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.644+0000 7f02d159a640 1 -- 172.21.15.114:0/2857193764 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2494939174 0 0) 0x7f02bc003520 con 0x7f02d41457f0 2024-03-29T11:30:40.642 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.644+0000 7f02d159a640 1 -- 172.21.15.114:0/2857193764 >> v1:172.21.15.114:6790/0 conn(0x7f02d40a9d70 legacy=0x7f02d414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:40.642 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.644+0000 7f02d159a640 1 -- 172.21.15.114:0/2857193764 >> v1:172.21.15.114:6789/0 conn(0x7f02d413f650 legacy=0x7f02d40b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:40.642 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.644+0000 7f02d159a640 10 monclient: _finish_hunting 0 2024-03-29T11:30:40.642 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.644+0000 7f02d159a640 1 monclient: found mon.noname-c 2024-03-29T11:30:40.642 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.644+0000 7f02d159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:40.642 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.644+0000 7f02d159a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.176:6789/0 2024-03-29T11:30:40.642 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.644+0000 7f02d159a640 1 -- 172.21.15.114:0/2857193764 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02d4146340 con 0x7f02d41457f0 2024-03-29T11:30:40.643 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.644+0000 7f02d159a640 10 monclient: _finish_auth 0 2024-03-29T11:30:40.643 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.644+0000 7f02d159a640 10 monclient: _check_auth_tickets 2024-03-29T11:30:40.643 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.644+0000 7f02d159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:40.643 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.644+0000 7f02d8cd9640 5 monclient: authenticate success, global_id 4549 2024-03-29T11:30:40.643 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.644+0000 7f02d8cd9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:30:40.643 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.644+0000 7f02d159a640 1 -- 172.21.15.114:0/2857193764 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f02cc001a10 con 0x7f02d41457f0 2024-03-29T11:30:40.643 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.644+0000 7f02d159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:40.643 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.644+0000 7f02d159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:30:40.643 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.644+0000 7f02d159a640 10 monclient: dump: 2024-03-29T11:30:40.643 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:30:40.643 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:30:40.643 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:30:40.643 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:30:40.643 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:40.643 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:30:40.644 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:30:40.644 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:30:40.644 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:30:40.644 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:30:40.644 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.644+0000 7f02d8cd9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:30:40.644 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.644+0000 7f02d159a640 1 -- 172.21.15.114:0/2857193764 <== mon.1 v1:172.21.15.176:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f02cc001c30 con 0x7f02d41457f0 2024-03-29T11:30:40.644 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.645+0000 7f02d159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:40.644 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.645+0000 7f02d8cd9640 10 monclient: get_monmap_and_config success 2024-03-29T11:30:40.644 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.645+0000 7f02d8cd9640 4 set_mon_vals no callback set 2024-03-29T11:30:40.644 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.645+0000 7f02d159a640 1 -- 172.21.15.114:0/2857193764 <== mon.1 v1:172.21.15.176:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f02cc001fc0 con 0x7f02d41457f0 2024-03-29T11:30:40.644 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.645+0000 7f02d159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:40.644 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.645+0000 7f02d159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:30:40.644 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.645+0000 7f02d159a640 10 monclient: dump: 2024-03-29T11:30:40.644 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:30:40.644 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:30:40.645 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:30:40.645 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:30:40.645 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:40.645 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:30:40.645 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:30:40.645 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:30:40.645 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:30:40.645 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:30:40.645 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.645+0000 7f02d8cd9640 10 monclient: shutdown 2024-03-29T11:30:40.645 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.645+0000 7f02d8cd9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:40.645 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.645+0000 7f02d8cd9640 1 -- 172.21.15.114:0/2857193764 >> v1:172.21.15.176:6789/0 conn(0x7f02d41457f0 legacy=0x7f02d413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:40.645 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.646+0000 7f02d8cd9640 1 -- 172.21.15.114:0/2857193764 shutdown_connections 2024-03-29T11:30:40.645 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.646+0000 7f02d8cd9640 1 -- 172.21.15.114:0/2857193764 >> 172.21.15.114:0/2857193764 conn(0x7f02d4066a00 msgr2=0x7f02d4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:40.645 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.646+0000 7f02d8cd9640 1 -- 172.21.15.114:0/2857193764 shutdown_connections 2024-03-29T11:30:40.645 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.646+0000 7f02d8cd9640 1 -- 172.21.15.114:0/2857193764 wait complete. 2024-03-29T11:30:40.646 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.647+0000 7f02d8cd9640 10 monclient: build_initial_monmap 2024-03-29T11:30:40.646 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.647+0000 7f02d8cd9640 10 monclient: monmap: 2024-03-29T11:30:40.646 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:30:40.646 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:30:40.646 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T11:30:40.646 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T11:30:40.646 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:40.646 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:30:40.646 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:30:40.646 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T11:30:40.646 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T11:30:40.646 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:30:40.646 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.648+0000 7f02d8cd9640 1 librados: starting msgr at 2024-03-29T11:30:40.646 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.648+0000 7f02d8cd9640 1 librados: starting objecter 2024-03-29T11:30:40.647 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.648+0000 7f02d8cd9640 1 Processor -- start 2024-03-29T11:30:40.647 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.648+0000 7f02d8cd9640 1 -- start start 2024-03-29T11:30:40.647 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.648+0000 7f02d8cd9640 1 librados: setting wanted keys 2024-03-29T11:30:40.647 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.648+0000 7f02d8cd9640 1 librados: calling monclient init 2024-03-29T11:30:40.647 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.648+0000 7f02d8cd9640 10 monclient: init 2024-03-29T11:30:40.653 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.649+0000 7f02d8cd9640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:40.653 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.649+0000 7f02d8cd9640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:30:40.653 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.649+0000 7f02d8cd9640 10 monclient(hunting): picked mon.noname-c con 0x7f02d40a9d70 addr v1:172.21.15.114:6790/0 2024-03-29T11:30:40.653 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.649+0000 7f02d8cd9640 10 monclient(hunting): picked mon.noname-b con 0x7f02d413f650 addr v1:172.21.15.176:6789/0 2024-03-29T11:30:40.653 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.649+0000 7f02d8cd9640 10 monclient(hunting): picked mon.noname-a con 0x7f02d41457f0 addr v1:172.21.15.114:6789/0 2024-03-29T11:30:40.653 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.649+0000 7f02d8cd9640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02d406e2c0 con 0x7f02d41457f0 2024-03-29T11:30:40.653 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.649+0000 7f02d8cd9640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02d406f720 con 0x7f02d413f650 2024-03-29T11:30:40.653 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.649+0000 7f02d8cd9640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02d406de90 con 0x7f02d40a9d70 2024-03-29T11:30:40.653 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.649+0000 7f02d259c640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f02d40a9d70 0x7f02d41df590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:57260/0 (socket says 172.21.15.114:57260) 2024-03-29T11:30:40.653 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.649+0000 7f02d259c640 1 -- 172.21.15.114:0/235469625 learned_addr learned my addr 172.21.15.114:0/235469625 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:30:40.653 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.649+0000 7f02d8cd9640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:40.654 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.649+0000 7f02c2ffd640 10 client.?.objecter ms_handle_connect 0x7f02d40a9d70 2024-03-29T11:30:40.654 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.649+0000 7f02c2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:40.654 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.649+0000 7f02d8cd9640 10 monclient(hunting): authenticate will time out at 1510.831918s 2024-03-29T11:30:40.654 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.650+0000 7f02c2ffd640 1 -- 172.21.15.114:0/235469625 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 120386853 0 0) 0x7f02d406de90 con 0x7f02d40a9d70 2024-03-29T11:30:40.654 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.650+0000 7f02c2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:30:40.654 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.650+0000 7f02c2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:40.654 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.650+0000 7f02c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:40.654 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.650+0000 7f02c2ffd640 10 monclient(hunting): my global_id is 4604 2024-03-29T11:30:40.654 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.650+0000 7f02c2ffd640 1 -- 172.21.15.114:0/235469625 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02b4004eb0 con 0x7f02d40a9d70 2024-03-29T11:30:40.654 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.650+0000 7f02c2ffd640 10 client.?.objecter ms_handle_connect 0x7f02d41457f0 2024-03-29T11:30:40.654 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.650+0000 7f02c2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:40.654 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.650+0000 7f02c2ffd640 10 client.?.objecter ms_handle_connect 0x7f02d413f650 2024-03-29T11:30:40.654 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.650+0000 7f02c2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:40.654 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.651+0000 7f02c2ffd640 1 -- 172.21.15.114:0/235469625 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1003263987 0 0) 0x7f02d406f720 con 0x7f02d413f650 2024-03-29T11:30:40.655 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.651+0000 7f02c2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:30:40.655 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.651+0000 7f02c2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:40.655 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.651+0000 7f02c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:40.655 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.651+0000 7f02c2ffd640 10 monclient(hunting): my global_id is 4552 2024-03-29T11:30:40.655 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.651+0000 7f02c2ffd640 1 -- 172.21.15.114:0/235469625 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02d406de90 con 0x7f02d413f650 2024-03-29T11:30:40.655 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.651+0000 7f02c2ffd640 1 -- 172.21.15.114:0/235469625 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1874472719 0 0) 0x7f02b4004eb0 con 0x7f02d40a9d70 2024-03-29T11:30:40.655 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.651+0000 7f02c2ffd640 1 -- 172.21.15.114:0/235469625 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f02d406f720 con 0x7f02d40a9d70 2024-03-29T11:30:40.655 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.651+0000 7f02c2ffd640 1 -- 172.21.15.114:0/235469625 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f02cc001f40 con 0x7f02d40a9d70 2024-03-29T11:30:40.655 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.651+0000 7f02c2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:40.655 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.651+0000 7f02c2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:30:40.655 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.651+0000 7f02c2ffd640 10 monclient(hunting): dump: 2024-03-29T11:30:40.655 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:30:40.655 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:30:40.655 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:30:40.656 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:30:40.656 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:40.656 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:30:40.656 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:30:40.656 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:30:40.656 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:30:40.656 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:30:40.656 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.651+0000 7f02c2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:30:40.656 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.651+0000 7f02c2ffd640 1 -- 172.21.15.114:0/235469625 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 130209786 0 0) 0x7f02d406e2c0 con 0x7f02d41457f0 2024-03-29T11:30:40.656 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.651+0000 7f02c2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:30:40.656 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.651+0000 7f02c2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:40.656 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.651+0000 7f02c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:40.656 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.651+0000 7f02c2ffd640 10 monclient(hunting): my global_id is 4569 2024-03-29T11:30:40.656 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.651+0000 7f02c2ffd640 1 -- 172.21.15.114:0/235469625 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02b4004eb0 con 0x7f02d41457f0 2024-03-29T11:30:40.656 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.651+0000 7f02c2ffd640 1 -- 172.21.15.114:0/235469625 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2389204236 0 0) 0x7f02d406de90 con 0x7f02d413f650 2024-03-29T11:30:40.657 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.651+0000 7f02c2ffd640 1 -- 172.21.15.114:0/235469625 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f02d406e2c0 con 0x7f02d413f650 2024-03-29T11:30:40.657 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.651+0000 7f02c2ffd640 1 -- 172.21.15.114:0/235469625 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f02c8001920 con 0x7f02d413f650 2024-03-29T11:30:40.657 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.652+0000 7f02c2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:40.657 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.652+0000 7f02c2ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:30:40.657 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.652+0000 7f02c2ffd640 10 monclient(hunting): dump: 2024-03-29T11:30:40.657 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:30:40.657 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:30:40.657 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:30:40.657 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:30:40.657 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:40.657 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:30:40.657 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:30:40.657 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:30:40.657 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:30:40.657 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:30:40.658 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.652+0000 7f02c2ffd640 1 -- 172.21.15.114:0/235469625 <== mon.1 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1620287320 0 0) 0x7f02d406e2c0 con 0x7f02d413f650 2024-03-29T11:30:40.658 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.652+0000 7f02c2ffd640 1 -- 172.21.15.114:0/235469625 >> v1:172.21.15.114:6790/0 conn(0x7f02d40a9d70 legacy=0x7f02d41df590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:40.658 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.652+0000 7f02c2ffd640 1 -- 172.21.15.114:0/235469625 >> v1:172.21.15.114:6789/0 conn(0x7f02d41457f0 legacy=0x7f02d41e6170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:40.658 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.652+0000 7f02c2ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:30:40.658 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.652+0000 7f02c2ffd640 1 monclient: found mon.b 2024-03-29T11:30:40.658 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.652+0000 7f02c2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:40.658 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.652+0000 7f02c2ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:30:40.658 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.652+0000 7f02c2ffd640 1 -- 172.21.15.114:0/235469625 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02d4142ca0 con 0x7f02d413f650 2024-03-29T11:30:40.658 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.652+0000 7f02c2ffd640 10 monclient: _finish_auth 0 2024-03-29T11:30:40.658 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.652+0000 7f02c2ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:30:40.658 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.652+0000 7f02c2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:40.658 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.652+0000 7f02d8cd9640 5 monclient: authenticate success, global_id 4552 2024-03-29T11:30:40.658 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.652+0000 7f02d8cd9640 10 monclient: _renew_subs 2024-03-29T11:30:40.658 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.652+0000 7f02d8cd9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:30:40.659 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.652+0000 7f02d8cd9640 1 -- 172.21.15.114:0/235469625 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02d4142270 con 0x7f02d413f650 2024-03-29T11:30:40.659 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.652+0000 7f02d8cd9640 10 client.4552.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:30:40.659 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.652+0000 7f02d8cd9640 10 monclient: _renew_subs 2024-03-29T11:30:40.659 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.652+0000 7f02d8cd9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:30:40.659 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.652+0000 7f02d8cd9640 1 -- 172.21.15.114:0/235469625 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f02d41df010 con 0x7f02d413f650 2024-03-29T11:30:40.659 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.652+0000 7f02d8cd9640 1 librados: init done 2024-03-29T11:30:40.659 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.653+0000 7f02d8cd9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:40.659 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.653+0000 7f02d8cd9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:40.659 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.653+0000 7f02d8cd9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:30:40.659 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.653+0000 7f02d8cd9640 1 -- 172.21.15.114:0/235469625 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02d406e020 con 0x7f02d413f650 2024-03-29T11:30:40.659 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.653+0000 7f02c2ffd640 1 -- 172.21.15.114:0/235469625 <== mon.1 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f02c8001d30 con 0x7f02d413f650 2024-03-29T11:30:40.659 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.653+0000 7f02c2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:40.659 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.653+0000 7f02d0d99640 4 set_mon_vals no callback set 2024-03-29T11:30:40.659 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.654+0000 7f02c2ffd640 1 -- 172.21.15.114:0/235469625 <== mon.1 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f02c80037b0 con 0x7f02d413f650 2024-03-29T11:30:40.659 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.654+0000 7f02c2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:40.660 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.654+0000 7f02c2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:30:40.660 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.654+0000 7f02c2ffd640 10 monclient: dump: 2024-03-29T11:30:40.660 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:30:40.660 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:30:40.660 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:30:40.660 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:30:40.660 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:40.660 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:30:40.660 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:30:40.660 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:30:40.660 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:30:40.660 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:30:40.660 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.656+0000 7f02c2ffd640 1 -- 172.21.15.114:0/235469625 <== mon.1 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1594292461 0 0) 0x7f02c8003990 con 0x7f02d413f650 2024-03-29T11:30:40.660 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.656+0000 7f02c2ffd640 1 -- 172.21.15.114:0/235469625 <== mon.1 v1:172.21.15.176:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3238987313 0 0) 0x7f02c80535b0 con 0x7f02d413f650 2024-03-29T11:30:40.661 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.656+0000 7f02c2ffd640 10 client.4552.objecter ms_dispatch 0x7f02d4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:30:40.661 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.656+0000 7f02c2ffd640 3 client.4552.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:30:40.661 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.656+0000 7f02d259c640 10 monclient: get_auth_request con 0x7f02b406dfa0 auth_method 0 2024-03-29T11:30:40.661 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.656+0000 7f02c2ffd640 3 client.4552.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:30:40.661 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.656+0000 7f02c2ffd640 20 client.4552.objecter dump_active .. 0 homeless 2024-03-29T11:30:40.661 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.657+0000 7f02c2ffd640 10 client.4552.objecter ms_handle_connect 0x7f02b406dfa0 2024-03-29T11:30:40.661 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.663+0000 7f02c2ffd640 1 -- 172.21.15.114:0/235469625 <== mon.1 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f02c804f850 con 0x7f02d413f650 2024-03-29T11:30:40.661 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.663+0000 7f02c2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:40.661 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:40.663+0000 7f02c2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:30:41.037 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:41.044+0000 7f02d8cd9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:41.037 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:41.044+0000 7f02d8cd9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:41.037 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:41.044+0000 7f02d8cd9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:30:41.037 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:41.044+0000 7f02d8cd9640 1 -- 172.21.15.114:0/235469625 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f02d4075f10 con 0x7f02d413f650 2024-03-29T11:30:41.038 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:41.045+0000 7f02c2ffd640 1 -- 172.21.15.114:0/235469625 <== mon.1 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2858224613) 0x7f02c804f850 con 0x7f02d413f650 2024-03-29T11:30:41.038 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:41.045+0000 7f02c2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:41.038 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:41.045+0000 7f02c2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:30:41.038 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T11:30:41.039 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"b0de2b9b-c45a-44c7-b6ee-e83b39a1e108","created":"2024-03-29T11:20:49.420660+0000","modified":"2024-03-29T11:20:51.700167+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:51.700167+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a64785d3-6b11-499a-b706-cc827ab96a74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5882dadb-e7a8-4604-ba9a-418013c106c8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d7e8c409-bc66-4e4a-bbdb-420bad31388e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0682989-3595-4cd6-90a5-ab9111c278e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"67068120-e3ff-4817-bd06-44feb7479a35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09e8b-8a85-4997-bdea-75d754e6ee67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2f0d9ead-f47f-4cb3-92a4-0a5ad23cd831","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fdb6a25f-6bd8-4c65-9b9b-b460ba7673a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:30:41.041 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:41.048+0000 7f02d8cd9640 10 librados: watch_flush enter 2024-03-29T11:30:41.043 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:41.049+0000 7f02d8cd9640 10 librados: watch_flush exit 2024-03-29T11:30:41.043 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:41.049+0000 7f02d8cd9640 20 client.4552.objecter shutdown clearing up homeless session... 2024-03-29T11:30:41.043 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:41.049+0000 7f02d8cd9640 10 client.4552.objecter successfully canceled tick 2024-03-29T11:30:41.043 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:41.049+0000 7f02d8cd9640 1 -- 172.21.15.114:0/235469625 >> v1:172.21.15.114:6816/4038755055 conn(0x7f02b406dfa0 legacy=0x7f02b4070460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:41.043 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:41.049+0000 7f02d8cd9640 10 monclient: shutdown 2024-03-29T11:30:41.043 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:41.049+0000 7f02d8cd9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:41.043 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:41.049+0000 7f02d8cd9640 1 -- 172.21.15.114:0/235469625 >> v1:172.21.15.176:6789/0 conn(0x7f02d413f650 legacy=0x7f02d413fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:41.043 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:41.049+0000 7f02d8cd9640 1 -- 172.21.15.114:0/235469625 shutdown_connections 2024-03-29T11:30:41.043 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:41.049+0000 7f02d8cd9640 1 -- 172.21.15.114:0/235469625 >> 172.21.15.114:0/235469625 conn(0x7f02d4066a00 msgr2=0x7f02d4066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:41.043 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:41.050+0000 7f02d8cd9640 1 -- 172.21.15.114:0/235469625 shutdown_connections 2024-03-29T11:30:41.043 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:41.050+0000 7f02d8cd9640 1 -- 172.21.15.114:0/235469625 wait complete. 2024-03-29T11:30:41.043 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:41.050+0000 7f02d8cd9640 1 librados: shutdown 2024-03-29T11:30:41.058 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:30:47.865 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:30:47.999 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.006+0000 7f0dfed4b640 10 monclient: get_monmap_and_config 2024-03-29T11:30:48.000 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.006+0000 7f0dfed4b640 10 monclient: build_initial_monmap 2024-03-29T11:30:48.000 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.006+0000 7f0dfed4b640 10 monclient: monmap: 2024-03-29T11:30:48.000 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:30:48.000 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:30:48.000 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:30:48.007587+0000 2024-03-29T11:30:48.000 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:30:48.007587+0000 2024-03-29T11:30:48.000 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:48.000 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:30:48.000 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:30:48.000 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T11:30:48.000 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T11:30:48.000 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:30:48.003 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.007+0000 7f0dfed4b640 1 Processor -- start 2024-03-29T11:30:48.003 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.007+0000 7f0dfed4b640 1 -- start start 2024-03-29T11:30:48.003 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.007+0000 7f0dfed4b640 10 monclient: init 2024-03-29T11:30:48.003 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.008+0000 7f0dfed4b640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:48.003 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.008+0000 7f0dfed4b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:30:48.003 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.008+0000 7f0dfed4b640 10 monclient(hunting): picked mon.noname-a con 0x7f0df81422b0 addr v1:172.21.15.114:6789/0 2024-03-29T11:30:48.003 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.008+0000 7f0dfed4b640 10 monclient(hunting): picked mon.noname-c con 0x7f0df8142690 addr v1:172.21.15.176:6789/0 2024-03-29T11:30:48.003 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.008+0000 7f0dfd2e8640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f0df80b17d0 0x7f0df80b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:35400/0 (socket says 172.21.15.114:35400) 2024-03-29T11:30:48.003 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.008+0000 7f0dfd2e8640 1 -- 172.21.15.114:0/793808813 learned_addr learned my addr 172.21.15.114:0/793808813 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:30:48.003 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.008+0000 7f0dfed4b640 10 monclient(hunting): picked mon.noname-b con 0x7f0df80b17d0 addr v1:172.21.15.114:6790/0 2024-03-29T11:30:48.003 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.008+0000 7f0dfed4b640 1 -- 172.21.15.114:0/793808813 --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0df8142c80 con 0x7f0df81422b0 2024-03-29T11:30:48.003 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.008+0000 7f0dfed4b640 1 -- 172.21.15.114:0/793808813 --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0df80aae40 con 0x7f0df8142690 2024-03-29T11:30:48.003 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.008+0000 7f0dfed4b640 1 -- 172.21.15.114:0/793808813 --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0df813fbd0 con 0x7f0df80b17d0 2024-03-29T11:30:48.004 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.008+0000 7f0dfed4b640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:48.004 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.008+0000 7f0dfed4b640 10 monclient(hunting): authenticate will time out at 1518.190834s 2024-03-29T11:30:48.004 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.008+0000 7f0df77fe640 1 -- 172.21.15.114:0/793808813 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2320862350 0 0) 0x7f0df813fbd0 con 0x7f0df80b17d0 2024-03-29T11:30:48.004 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.008+0000 7f0df77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:30:48.004 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.008+0000 7f0df77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:48.004 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.008+0000 7f0df77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:48.004 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.008+0000 7f0df77fe640 10 monclient(hunting): my global_id is 4607 2024-03-29T11:30:48.004 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.008+0000 7f0df77fe640 1 -- 172.21.15.114:0/793808813 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0de8003520 con 0x7f0df80b17d0 2024-03-29T11:30:48.004 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.009+0000 7f0df77fe640 1 -- 172.21.15.114:0/793808813 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3129331064 0 0) 0x7f0de8003520 con 0x7f0df80b17d0 2024-03-29T11:30:48.004 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.009+0000 7f0df77fe640 1 -- 172.21.15.114:0/793808813 >> v1:172.21.15.176:6789/0 conn(0x7f0df8142690 legacy=0x7f0df80b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:30:48.004 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.009+0000 7f0df77fe640 1 -- 172.21.15.114:0/793808813 >> v1:172.21.15.114:6789/0 conn(0x7f0df81422b0 legacy=0x7f0df813f4c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:30:48.007 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.011+0000 7f0df77fe640 10 monclient: _finish_hunting 0 2024-03-29T11:30:48.007 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.011+0000 7f0df77fe640 1 monclient: found mon.noname-b 2024-03-29T11:30:48.007 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.011+0000 7f0df77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:48.007 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.011+0000 7f0df77fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.114:6790/0 2024-03-29T11:30:48.007 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.011+0000 7f0df77fe640 1 -- 172.21.15.114:0/793808813 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0df8150930 con 0x7f0df80b17d0 2024-03-29T11:30:48.007 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.011+0000 7f0df77fe640 10 monclient: _finish_auth 0 2024-03-29T11:30:48.007 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.011+0000 7f0df77fe640 10 monclient: _check_auth_tickets 2024-03-29T11:30:48.007 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.011+0000 7f0df77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:48.007 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.011+0000 7f0df77fe640 1 -- 172.21.15.114:0/793808813 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f0df0001a10 con 0x7f0df80b17d0 2024-03-29T11:30:48.007 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.011+0000 7f0df77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:48.007 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.011+0000 7f0df77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:30:48.007 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.011+0000 7f0df77fe640 10 monclient: dump: 2024-03-29T11:30:48.008 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:30:48.008 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:30:48.008 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:30:48.008 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:30:48.008 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:48.008 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:30:48.008 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:30:48.008 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:30:48.008 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:30:48.008 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:30:48.008 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.011+0000 7f0dfed4b640 5 monclient: authenticate success, global_id 4607 2024-03-29T11:30:48.008 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.011+0000 7f0dfed4b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:30:48.008 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.012+0000 7f0df77fe640 1 -- 172.21.15.114:0/793808813 <== mon.2 v1:172.21.15.114:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0df0001b70 con 0x7f0df80b17d0 2024-03-29T11:30:48.008 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.012+0000 7f0df77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:48.009 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.012+0000 7f0df77fe640 1 -- 172.21.15.114:0/793808813 <== mon.2 v1:172.21.15.114:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f0df0001fc0 con 0x7f0df80b17d0 2024-03-29T11:30:48.009 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.012+0000 7f0df77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:48.009 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.012+0000 7f0df77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:30:48.009 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.012+0000 7f0df77fe640 10 monclient: dump: 2024-03-29T11:30:48.009 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:30:48.009 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:30:48.009 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:30:48.009 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:30:48.009 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:48.009 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:30:48.009 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:30:48.009 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:30:48.009 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:30:48.009 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:30:48.009 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.012+0000 7f0dfed4b640 10 monclient: get_monmap_and_config success 2024-03-29T11:30:48.010 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.012+0000 7f0dfed4b640 4 set_mon_vals no callback set 2024-03-29T11:30:48.010 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.012+0000 7f0dfed4b640 10 monclient: shutdown 2024-03-29T11:30:48.010 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.012+0000 7f0dfed4b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:48.010 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.012+0000 7f0dfed4b640 1 -- 172.21.15.114:0/793808813 >> v1:172.21.15.114:6790/0 conn(0x7f0df80b17d0 legacy=0x7f0df80b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:48.010 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.013+0000 7f0dfed4b640 1 -- 172.21.15.114:0/793808813 shutdown_connections 2024-03-29T11:30:48.010 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.013+0000 7f0dfed4b640 1 -- 172.21.15.114:0/793808813 >> 172.21.15.114:0/793808813 conn(0x7f0df8066a00 msgr2=0x7f0df8066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:48.010 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.013+0000 7f0dfed4b640 1 -- 172.21.15.114:0/793808813 shutdown_connections 2024-03-29T11:30:48.010 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.013+0000 7f0dfed4b640 1 -- 172.21.15.114:0/793808813 wait complete. 2024-03-29T11:30:48.015 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.014+0000 7f0dfed4b640 10 monclient: build_initial_monmap 2024-03-29T11:30:48.016 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.015+0000 7f0dfed4b640 10 monclient: monmap: 2024-03-29T11:30:48.016 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:30:48.016 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:30:48.016 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T11:30:48.016 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T11:30:48.016 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:48.016 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:30:48.016 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:30:48.016 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T11:30:48.016 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T11:30:48.016 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:30:48.016 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.015+0000 7f0dfed4b640 1 librados: starting msgr at 2024-03-29T11:30:48.017 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.015+0000 7f0dfed4b640 1 librados: starting objecter 2024-03-29T11:30:48.017 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.015+0000 7f0dfed4b640 1 Processor -- start 2024-03-29T11:30:48.017 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.015+0000 7f0dfed4b640 1 -- start start 2024-03-29T11:30:48.017 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.015+0000 7f0dfed4b640 1 librados: setting wanted keys 2024-03-29T11:30:48.017 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.015+0000 7f0dfed4b640 1 librados: calling monclient init 2024-03-29T11:30:48.017 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.015+0000 7f0dfed4b640 10 monclient: init 2024-03-29T11:30:48.017 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.016+0000 7f0dfed4b640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:48.017 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.016+0000 7f0dfed4b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:30:48.017 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.016+0000 7f0dfed4b640 10 monclient(hunting): picked mon.noname-a con 0x7f0df80b17d0 addr v1:172.21.15.114:6789/0 2024-03-29T11:30:48.017 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.017+0000 7f0dfed4b640 10 monclient(hunting): picked mon.noname-b con 0x7f0df81422b0 addr v1:172.21.15.176:6789/0 2024-03-29T11:30:48.017 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.017+0000 7f0dfed4b640 10 monclient(hunting): picked mon.noname-c con 0x7f0df8142690 addr v1:172.21.15.114:6790/0 2024-03-29T11:30:48.017 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.017+0000 7f0dfed4b640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0df814cc90 con 0x7f0df80b17d0 2024-03-29T11:30:48.017 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.017+0000 7f0dfed4b640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0df814d500 con 0x7f0df81422b0 2024-03-29T11:30:48.017 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.017+0000 7f0dfed4b640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0df806e260 con 0x7f0df8142690 2024-03-29T11:30:48.017 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.017+0000 7f0dfed4b640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:48.017 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.017+0000 7f0dfed4b640 10 monclient(hunting): authenticate will time out at 1518.199363s 2024-03-29T11:30:48.018 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.017+0000 7f0dfd2e8640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f0df8142690 0x7f0df81e6620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:35930/0 (socket says 172.21.15.114:35930) 2024-03-29T11:30:48.018 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.017+0000 7f0dfd2e8640 1 -- 172.21.15.114:0/756061688 learned_addr learned my addr 172.21.15.114:0/756061688 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:30:48.018 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.017+0000 7f0df57fa640 10 client.?.objecter ms_handle_connect 0x7f0df8142690 2024-03-29T11:30:48.018 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.017+0000 7f0df57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:48.018 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.017+0000 7f0df57fa640 1 -- 172.21.15.114:0/756061688 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1202694018 0 0) 0x7f0df806e260 con 0x7f0df8142690 2024-03-29T11:30:48.018 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.017+0000 7f0df57fa640 10 monclient(hunting): _negotiate 2024-03-29T11:30:48.018 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.017+0000 7f0df57fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:48.018 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.017+0000 7f0df57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:48.018 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.017+0000 7f0df57fa640 10 monclient(hunting): my global_id is 4610 2024-03-29T11:30:48.018 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.017+0000 7f0df57fa640 1 -- 172.21.15.114:0/756061688 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ddc004eb0 con 0x7f0df8142690 2024-03-29T11:30:48.018 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.017+0000 7f0df57fa640 1 -- 172.21.15.114:0/756061688 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 469634432 0 0) 0x7f0ddc004eb0 con 0x7f0df8142690 2024-03-29T11:30:48.018 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.017+0000 7f0df57fa640 1 -- 172.21.15.114:0/756061688 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0df806e260 con 0x7f0df8142690 2024-03-29T11:30:48.018 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.017+0000 7f0df57fa640 1 -- 172.21.15.114:0/756061688 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f0df0001f40 con 0x7f0df8142690 2024-03-29T11:30:48.019 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.017+0000 7f0df57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:48.019 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.018+0000 7f0df57fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:30:48.019 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.018+0000 7f0df57fa640 10 monclient(hunting): dump: 2024-03-29T11:30:48.019 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:30:48.019 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:30:48.019 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:30:48.019 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:30:48.019 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:48.019 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:30:48.019 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:30:48.019 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:30:48.019 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:30:48.019 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:30:48.019 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.018+0000 7f0df57fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:30:48.020 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.018+0000 7f0df57fa640 1 -- 172.21.15.114:0/756061688 <== mon.2 v1:172.21.15.114:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 17431612 0 0) 0x7f0df806e260 con 0x7f0df8142690 2024-03-29T11:30:48.020 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.018+0000 7f0df57fa640 1 -- 172.21.15.114:0/756061688 >> v1:172.21.15.176:6789/0 conn(0x7f0df81422b0 legacy=0x7f0df81df080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:48.020 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.018+0000 7f0df57fa640 1 -- 172.21.15.114:0/756061688 >> v1:172.21.15.114:6789/0 conn(0x7f0df80b17d0 legacy=0x7f0df814c580 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:30:48.020 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.019+0000 7f0df57fa640 10 monclient: _finish_hunting 0 2024-03-29T11:30:48.020 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.019+0000 7f0df57fa640 1 monclient: found mon.c 2024-03-29T11:30:48.020 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.019+0000 7f0df57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:48.020 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.019+0000 7f0df57fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:30:48.020 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.019+0000 7f0df57fa640 1 -- 172.21.15.114:0/756061688 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0df814ce50 con 0x7f0df8142690 2024-03-29T11:30:48.020 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.019+0000 7f0df57fa640 10 monclient: _finish_auth 0 2024-03-29T11:30:48.020 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.019+0000 7f0df57fa640 10 monclient: _check_auth_tickets 2024-03-29T11:30:48.020 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.019+0000 7f0df57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:48.020 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.019+0000 7f0dfed4b640 5 monclient: authenticate success, global_id 4610 2024-03-29T11:30:48.020 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.019+0000 7f0df57fa640 1 -- 172.21.15.114:0/756061688 <== mon.2 v1:172.21.15.114:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0df00020a0 con 0x7f0df8142690 2024-03-29T11:30:48.021 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.019+0000 7f0dfed4b640 10 monclient: _renew_subs 2024-03-29T11:30:48.021 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.019+0000 7f0dfed4b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:30:48.021 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.019+0000 7f0dfed4b640 1 -- 172.21.15.114:0/756061688 --> v1:172.21.15.114:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0df814d100 con 0x7f0df8142690 2024-03-29T11:30:48.021 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.020+0000 7f0dfed4b640 10 client.4610.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:30:48.021 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.020+0000 7f0df57fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:48.021 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.020+0000 7f0df57fa640 1 -- 172.21.15.114:0/756061688 <== mon.2 v1:172.21.15.114:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f0df0005190 con 0x7f0df8142690 2024-03-29T11:30:48.021 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.020+0000 7f0df57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:48.021 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.020+0000 7f0df57fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:30:48.021 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.020+0000 7f0df57fa640 10 monclient: dump: 2024-03-29T11:30:48.021 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:30:48.021 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:30:48.021 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:30:48.021 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:30:48.021 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:48.021 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:30:48.022 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:30:48.022 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:30:48.022 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:30:48.022 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:30:48.022 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.020+0000 7f0dfed4b640 10 monclient: _renew_subs 2024-03-29T11:30:48.022 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.020+0000 7f0dfed4b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:30:48.022 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.020+0000 7f0dfed4b640 1 -- 172.21.15.114:0/756061688 --> v1:172.21.15.114:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0df81df970 con 0x7f0df8142690 2024-03-29T11:30:48.022 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.020+0000 7f0dfed4b640 1 librados: init done 2024-03-29T11:30:48.022 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.020+0000 7f0df6ffd640 4 set_mon_vals no callback set 2024-03-29T11:30:48.022 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.022+0000 7f0dfed4b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:48.022 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.022+0000 7f0dfed4b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:48.022 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.022+0000 7f0dfed4b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:30:48.022 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.022+0000 7f0dfed4b640 1 -- 172.21.15.114:0/756061688 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0dd8005080 con 0x7f0df8142690 2024-03-29T11:30:48.023 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.022+0000 7f0df57fa640 1 -- 172.21.15.114:0/756061688 <== mon.2 v1:172.21.15.114:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1594292461 0 0) 0x7f0df001b1e0 con 0x7f0df8142690 2024-03-29T11:30:48.024 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.026+0000 7f0dfcae7640 10 monclient: get_auth_request con 0x7f0ddc06d9c0 auth_method 0 2024-03-29T11:30:48.024 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.026+0000 7f0df57fa640 1 -- 172.21.15.114:0/756061688 <== mon.2 v1:172.21.15.114:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3238987313 0 0) 0x7f0df0085090 con 0x7f0df8142690 2024-03-29T11:30:48.024 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.026+0000 7f0df57fa640 10 client.4610.objecter ms_dispatch 0x7f0df8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:30:48.024 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.026+0000 7f0df57fa640 3 client.4610.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:30:48.024 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.026+0000 7f0df57fa640 3 client.4610.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:30:48.024 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.026+0000 7f0df57fa640 20 client.4610.objecter dump_active .. 0 homeless 2024-03-29T11:30:48.024 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.026+0000 7f0df57fa640 10 client.4610.objecter ms_handle_connect 0x7f0ddc06d9c0 2024-03-29T11:30:48.024 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.030+0000 7f0df57fa640 1 -- 172.21.15.114:0/756061688 <== mon.2 v1:172.21.15.114:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0df0050970 con 0x7f0df8142690 2024-03-29T11:30:48.024 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.030+0000 7f0df57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:48.024 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.030+0000 7f0df57fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:30:48.331 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.338+0000 7f0dfed4b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:48.331 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.338+0000 7f0dfed4b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:48.331 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.338+0000 7f0dfed4b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:30:48.331 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.338+0000 7f0dfed4b640 1 -- 172.21.15.114:0/756061688 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0dd8005600 con 0x7f0df8142690 2024-03-29T11:30:48.332 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.339+0000 7f0df57fa640 1 -- 172.21.15.114:0/756061688 <== mon.2 v1:172.21.15.114:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2858224613) 0x7f0df00528a0 con 0x7f0df8142690 2024-03-29T11:30:48.332 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.339+0000 7f0df57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:48.332 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.339+0000 7f0df57fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:30:48.332 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T11:30:48.332 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":10,"fsid":"b0de2b9b-c45a-44c7-b6ee-e83b39a1e108","created":"2024-03-29T11:20:49.420660+0000","modified":"2024-03-29T11:20:51.700167+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:51.700167+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a64785d3-6b11-499a-b706-cc827ab96a74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5882dadb-e7a8-4604-ba9a-418013c106c8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d7e8c409-bc66-4e4a-bbdb-420bad31388e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0682989-3595-4cd6-90a5-ab9111c278e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"67068120-e3ff-4817-bd06-44feb7479a35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09e8b-8a85-4997-bdea-75d754e6ee67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2f0d9ead-f47f-4cb3-92a4-0a5ad23cd831","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fdb6a25f-6bd8-4c65-9b9b-b460ba7673a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:30:48.336 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.342+0000 7f0dfed4b640 10 librados: watch_flush enter 2024-03-29T11:30:48.336 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.342+0000 7f0dfed4b640 10 librados: watch_flush exit 2024-03-29T11:30:48.336 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.342+0000 7f0dfed4b640 20 client.4610.objecter shutdown clearing up homeless session... 2024-03-29T11:30:48.336 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.342+0000 7f0dfed4b640 10 client.4610.objecter successfully canceled tick 2024-03-29T11:30:48.337 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.342+0000 7f0dfed4b640 1 -- 172.21.15.114:0/756061688 >> v1:172.21.15.114:6816/4038755055 conn(0x7f0ddc06d9c0 legacy=0x7f0ddc06fe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:48.337 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.343+0000 7f0dfed4b640 10 monclient: shutdown 2024-03-29T11:30:48.337 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.343+0000 7f0dfed4b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:48.337 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.343+0000 7f0dfed4b640 1 -- 172.21.15.114:0/756061688 >> v1:172.21.15.114:6790/0 conn(0x7f0df8142690 legacy=0x7f0df81e6620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:48.337 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.343+0000 7f0dfed4b640 1 -- 172.21.15.114:0/756061688 shutdown_connections 2024-03-29T11:30:48.337 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.343+0000 7f0dfed4b640 1 -- 172.21.15.114:0/756061688 >> 172.21.15.114:0/756061688 conn(0x7f0df8066a00 msgr2=0x7f0df814f2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:48.337 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.343+0000 7f0dfed4b640 1 -- 172.21.15.114:0/756061688 shutdown_connections 2024-03-29T11:30:48.337 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.343+0000 7f0dfed4b640 1 -- 172.21.15.114:0/756061688 wait complete. 2024-03-29T11:30:48.337 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:48.343+0000 7f0dfed4b640 1 librados: shutdown 2024-03-29T11:30:48.350 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:30:55.158 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:30:55.305 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.312+0000 7fdd66d5e640 10 monclient: get_monmap_and_config 2024-03-29T11:30:55.306 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.312+0000 7fdd66d5e640 10 monclient: build_initial_monmap 2024-03-29T11:30:55.306 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.312+0000 7fdd66d5e640 10 monclient: monmap: 2024-03-29T11:30:55.306 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:30:55.306 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:30:55.306 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:30:55.313176+0000 2024-03-29T11:30:55.306 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:30:55.313176+0000 2024-03-29T11:30:55.306 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:55.306 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:30:55.306 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:30:55.306 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T11:30:55.306 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T11:30:55.306 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:30:55.307 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.313+0000 7fdd66d5e640 1 Processor -- start 2024-03-29T11:30:55.307 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.313+0000 7fdd66d5e640 1 -- start start 2024-03-29T11:30:55.307 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.313+0000 7fdd66d5e640 10 monclient: init 2024-03-29T11:30:55.308 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.314+0000 7fdd66d5e640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:55.308 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.314+0000 7fdd66d5e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:30:55.308 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.314+0000 7fdd66d5e640 10 monclient(hunting): picked mon.noname-a con 0x7fdd601422b0 addr v1:172.21.15.114:6789/0 2024-03-29T11:30:55.308 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.314+0000 7fdd66d5e640 10 monclient(hunting): picked mon.noname-c con 0x7fdd60142690 addr v1:172.21.15.176:6789/0 2024-03-29T11:30:55.308 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.314+0000 7fdd66d5e640 10 monclient(hunting): picked mon.noname-b con 0x7fdd600b17d0 addr v1:172.21.15.114:6790/0 2024-03-29T11:30:55.308 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.314+0000 7fdd66d5e640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd60142c80 con 0x7fdd601422b0 2024-03-29T11:30:55.308 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.314+0000 7fdd66d5e640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd600aae40 con 0x7fdd60142690 2024-03-29T11:30:55.308 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.314+0000 7fdd66d5e640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd6013fbd0 con 0x7fdd600b17d0 2024-03-29T11:30:55.309 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.314+0000 7fdd66d5e640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:55.309 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.314+0000 7fdd66d5e640 10 monclient(hunting): authenticate will time out at 1525.496500s 2024-03-29T11:30:55.311 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.317+0000 7fdd652fb640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7fdd600b17d0 0x7fdd600b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:38778/0 (socket says 172.21.15.114:38778) 2024-03-29T11:30:55.311 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.317+0000 7fdd652fb640 1 -- 172.21.15.114:0/436660539 learned_addr learned my addr 172.21.15.114:0/436660539 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:30:55.311 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.317+0000 7fdd5f7fe640 1 -- 172.21.15.114:0/436660539 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 821387433 0 0) 0x7fdd6013fbd0 con 0x7fdd600b17d0 2024-03-29T11:30:55.311 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.317+0000 7fdd5f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:30:55.311 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.317+0000 7fdd5f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:55.311 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.317+0000 7fdd5f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:55.311 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.317+0000 7fdd5f7fe640 10 monclient(hunting): my global_id is 4613 2024-03-29T11:30:55.312 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.317+0000 7fdd5f7fe640 1 -- 172.21.15.114:0/436660539 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd4c003520 con 0x7fdd600b17d0 2024-03-29T11:30:55.312 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.318+0000 7fdd5f7fe640 1 -- 172.21.15.114:0/436660539 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2880644145 0 0) 0x7fdd4c003520 con 0x7fdd600b17d0 2024-03-29T11:30:55.312 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.318+0000 7fdd5f7fe640 1 -- 172.21.15.114:0/436660539 >> v1:172.21.15.176:6789/0 conn(0x7fdd60142690 legacy=0x7fdd600b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:30:55.312 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.318+0000 7fdd5f7fe640 1 -- 172.21.15.114:0/436660539 >> v1:172.21.15.114:6789/0 conn(0x7fdd601422b0 legacy=0x7fdd6013f4c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:30:55.312 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.318+0000 7fdd5f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:30:55.312 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.318+0000 7fdd5f7fe640 1 monclient: found mon.noname-b 2024-03-29T11:30:55.312 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.318+0000 7fdd5f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:55.312 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.318+0000 7fdd5f7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.114:6790/0 2024-03-29T11:30:55.312 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.318+0000 7fdd5f7fe640 1 -- 172.21.15.114:0/436660539 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd60151940 con 0x7fdd600b17d0 2024-03-29T11:30:55.312 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.318+0000 7fdd5f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:30:55.313 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.318+0000 7fdd5f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:30:55.313 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.318+0000 7fdd5f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:55.313 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.318+0000 7fdd5f7fe640 1 -- 172.21.15.114:0/436660539 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fdd58001690 con 0x7fdd600b17d0 2024-03-29T11:30:55.313 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.318+0000 7fdd5f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:55.313 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.318+0000 7fdd5f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:30:55.313 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.318+0000 7fdd5f7fe640 10 monclient: dump: 2024-03-29T11:30:55.313 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:30:55.313 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:30:55.313 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:30:55.313 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:30:55.313 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:55.313 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:30:55.313 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:30:55.313 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:30:55.313 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:30:55.314 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:30:55.314 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.320+0000 7fdd66d5e640 5 monclient: authenticate success, global_id 4613 2024-03-29T11:30:55.314 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.320+0000 7fdd66d5e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:30:55.314 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.320+0000 7fdd5f7fe640 1 -- 172.21.15.114:0/436660539 <== mon.2 v1:172.21.15.114:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdd580018f0 con 0x7fdd600b17d0 2024-03-29T11:30:55.314 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.320+0000 7fdd5f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:55.314 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.320+0000 7fdd5f7fe640 1 -- 172.21.15.114:0/436660539 <== mon.2 v1:172.21.15.114:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fdd58001c80 con 0x7fdd600b17d0 2024-03-29T11:30:55.314 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.320+0000 7fdd5f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:55.314 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.320+0000 7fdd5f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:30:55.314 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.320+0000 7fdd5f7fe640 10 monclient: dump: 2024-03-29T11:30:55.315 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:30:55.315 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:30:55.315 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:30:55.315 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:30:55.315 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:55.315 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:30:55.315 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:30:55.315 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:30:55.315 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:30:55.315 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:30:55.315 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.320+0000 7fdd66d5e640 10 monclient: get_monmap_and_config success 2024-03-29T11:30:55.315 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.320+0000 7fdd66d5e640 4 set_mon_vals no callback set 2024-03-29T11:30:55.316 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.321+0000 7fdd66d5e640 10 monclient: shutdown 2024-03-29T11:30:55.316 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.321+0000 7fdd66d5e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:55.316 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.321+0000 7fdd66d5e640 1 -- 172.21.15.114:0/436660539 >> v1:172.21.15.114:6790/0 conn(0x7fdd600b17d0 legacy=0x7fdd600b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:55.316 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.321+0000 7fdd66d5e640 1 -- 172.21.15.114:0/436660539 shutdown_connections 2024-03-29T11:30:55.316 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.321+0000 7fdd66d5e640 1 -- 172.21.15.114:0/436660539 >> 172.21.15.114:0/436660539 conn(0x7fdd60066a00 msgr2=0x7fdd60066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:55.316 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.322+0000 7fdd66d5e640 1 -- 172.21.15.114:0/436660539 shutdown_connections 2024-03-29T11:30:55.317 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.322+0000 7fdd66d5e640 1 -- 172.21.15.114:0/436660539 wait complete. 2024-03-29T11:30:55.317 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.324+0000 7fdd66d5e640 10 monclient: build_initial_monmap 2024-03-29T11:30:55.317 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.324+0000 7fdd66d5e640 10 monclient: monmap: 2024-03-29T11:30:55.317 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:30:55.318 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:30:55.318 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T11:30:55.318 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T11:30:55.318 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:55.318 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:30:55.318 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:30:55.318 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T11:30:55.318 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T11:30:55.318 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:30:55.319 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.325+0000 7fdd66d5e640 1 librados: starting msgr at 2024-03-29T11:30:55.319 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.325+0000 7fdd66d5e640 1 librados: starting objecter 2024-03-29T11:30:55.319 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.325+0000 7fdd66d5e640 1 Processor -- start 2024-03-29T11:30:55.319 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.325+0000 7fdd66d5e640 1 -- start start 2024-03-29T11:30:55.319 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.325+0000 7fdd66d5e640 1 librados: setting wanted keys 2024-03-29T11:30:55.323 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.325+0000 7fdd66d5e640 1 librados: calling monclient init 2024-03-29T11:30:55.323 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.325+0000 7fdd66d5e640 10 monclient: init 2024-03-29T11:30:55.324 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.327+0000 7fdd66d5e640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:55.324 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.327+0000 7fdd66d5e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:30:55.324 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.327+0000 7fdd66d5e640 10 monclient(hunting): picked mon.noname-a con 0x7fdd600b17d0 addr v1:172.21.15.114:6789/0 2024-03-29T11:30:55.324 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.327+0000 7fdd66d5e640 10 monclient(hunting): picked mon.noname-c con 0x7fdd601422b0 addr v1:172.21.15.114:6790/0 2024-03-29T11:30:55.324 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.327+0000 7fdd66d5e640 10 monclient(hunting): picked mon.noname-b con 0x7fdd60142690 addr v1:172.21.15.176:6789/0 2024-03-29T11:30:55.324 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.327+0000 7fdd66d5e640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd60131e10 con 0x7fdd600b17d0 2024-03-29T11:30:55.324 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.327+0000 7fdd66d5e640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd6006ea20 con 0x7fdd60142690 2024-03-29T11:30:55.324 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.327+0000 7fdd66d5e640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd601322a0 con 0x7fdd601422b0 2024-03-29T11:30:55.324 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.327+0000 7fdd66d5e640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:55.324 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.327+0000 7fdd66d5e640 10 monclient(hunting): authenticate will time out at 1525.509984s 2024-03-29T11:30:55.324 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.328+0000 7fdd5ffff640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7fdd601422b0 0x7fdd6012fcc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:50942/0 (socket says 172.21.15.114:50942) 2024-03-29T11:30:55.324 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.328+0000 7fdd5ffff640 1 -- 172.21.15.114:0/2976632648 learned_addr learned my addr 172.21.15.114:0/2976632648 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:30:55.324 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.328+0000 7fdd5d7fa640 10 client.?.objecter ms_handle_connect 0x7fdd601422b0 2024-03-29T11:30:55.324 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.328+0000 7fdd5d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:55.324 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.328+0000 7fdd5d7fa640 1 -- 172.21.15.114:0/2976632648 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1474551052 0 0) 0x7fdd601322a0 con 0x7fdd601422b0 2024-03-29T11:30:55.325 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.328+0000 7fdd5d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:30:55.325 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.328+0000 7fdd5d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:55.325 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.328+0000 7fdd5d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:55.325 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.328+0000 7fdd5d7fa640 10 monclient(hunting): my global_id is 4616 2024-03-29T11:30:55.325 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.329+0000 7fdd5d7fa640 1 -- 172.21.15.114:0/2976632648 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd40004eb0 con 0x7fdd601422b0 2024-03-29T11:30:55.325 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.329+0000 7fdd5d7fa640 1 -- 172.21.15.114:0/2976632648 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2254084793 0 0) 0x7fdd40004eb0 con 0x7fdd601422b0 2024-03-29T11:30:55.325 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.329+0000 7fdd5d7fa640 1 -- 172.21.15.114:0/2976632648 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdd601322a0 con 0x7fdd601422b0 2024-03-29T11:30:55.325 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.329+0000 7fdd5d7fa640 1 -- 172.21.15.114:0/2976632648 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fdd500017d0 con 0x7fdd601422b0 2024-03-29T11:30:55.325 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.329+0000 7fdd5d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:55.325 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.329+0000 7fdd5d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:30:55.325 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.329+0000 7fdd5d7fa640 10 monclient(hunting): dump: 2024-03-29T11:30:55.325 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:30:55.325 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:30:55.325 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:30:55.326 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:30:55.326 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:55.326 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:30:55.326 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:30:55.326 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:30:55.326 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:30:55.326 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:30:55.326 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.329+0000 7fdd5d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:30:55.326 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.329+0000 7fdd5d7fa640 1 -- 172.21.15.114:0/2976632648 <== mon.2 v1:172.21.15.114:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1808453419 0 0) 0x7fdd601322a0 con 0x7fdd601422b0 2024-03-29T11:30:55.326 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.329+0000 7fdd5d7fa640 1 -- 172.21.15.114:0/2976632648 >> v1:172.21.15.176:6789/0 conn(0x7fdd60142690 legacy=0x7fdd601e6120 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:30:55.327 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.331+0000 7fdd5d7fa640 1 -- 172.21.15.114:0/2976632648 >> v1:172.21.15.114:6789/0 conn(0x7fdd600b17d0 legacy=0x7fdd60131520 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:30:55.327 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.332+0000 7fdd5d7fa640 10 monclient: _finish_hunting 0 2024-03-29T11:30:55.327 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.332+0000 7fdd5d7fa640 1 monclient: found mon.c 2024-03-29T11:30:55.327 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.332+0000 7fdd5d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:55.327 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.332+0000 7fdd5d7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:30:55.327 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.332+0000 7fdd5d7fa640 1 -- 172.21.15.114:0/2976632648 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd6006ec90 con 0x7fdd601422b0 2024-03-29T11:30:55.328 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.332+0000 7fdd5d7fa640 10 monclient: _finish_auth 0 2024-03-29T11:30:55.328 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.332+0000 7fdd5d7fa640 10 monclient: _check_auth_tickets 2024-03-29T11:30:55.328 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.332+0000 7fdd5d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:55.328 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.332+0000 7fdd66d5e640 5 monclient: authenticate success, global_id 4616 2024-03-29T11:30:55.328 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.332+0000 7fdd66d5e640 10 monclient: _renew_subs 2024-03-29T11:30:55.328 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.332+0000 7fdd66d5e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:30:55.328 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.332+0000 7fdd66d5e640 1 -- 172.21.15.114:0/2976632648 --> v1:172.21.15.114:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd60150fe0 con 0x7fdd601422b0 2024-03-29T11:30:55.328 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.332+0000 7fdd66d5e640 10 client.4616.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:30:55.328 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.332+0000 7fdd66d5e640 10 monclient: _renew_subs 2024-03-29T11:30:55.328 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.332+0000 7fdd66d5e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:30:55.328 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.332+0000 7fdd66d5e640 1 -- 172.21.15.114:0/2976632648 --> v1:172.21.15.114:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd601303d0 con 0x7fdd601422b0 2024-03-29T11:30:55.328 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.332+0000 7fdd66d5e640 1 librados: init done 2024-03-29T11:30:55.328 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.333+0000 7fdd5d7fa640 1 -- 172.21.15.114:0/2976632648 <== mon.2 v1:172.21.15.114:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdd50001a30 con 0x7fdd601422b0 2024-03-29T11:30:55.328 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.333+0000 7fdd5d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:55.329 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.333+0000 7fdd5d7fa640 1 -- 172.21.15.114:0/2976632648 <== mon.2 v1:172.21.15.114:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fdd50004ac0 con 0x7fdd601422b0 2024-03-29T11:30:55.329 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.333+0000 7fdd5d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:55.329 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.333+0000 7fdd5d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:30:55.329 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.333+0000 7fdd5d7fa640 10 monclient: dump: 2024-03-29T11:30:55.329 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:30:55.329 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:30:55.329 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:30:55.329 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:30:55.329 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:55.329 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:30:55.329 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:30:55.329 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:30:55.329 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:30:55.329 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:30:55.329 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.333+0000 7fdd5e7fc640 4 set_mon_vals no callback set 2024-03-29T11:30:55.330 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.335+0000 7fdd5d7fa640 1 -- 172.21.15.114:0/2976632648 <== mon.2 v1:172.21.15.114:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1594292461 0 0) 0x7fdd5001aa90 con 0x7fdd601422b0 2024-03-29T11:30:55.330 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.334+0000 7fdd66d5e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:55.330 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.336+0000 7fdd5d7fa640 1 -- 172.21.15.114:0/2976632648 <== mon.2 v1:172.21.15.114:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1864569634 0 0) 0x7fdd50084b50 con 0x7fdd601422b0 2024-03-29T11:30:55.330 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.336+0000 7fdd5d7fa640 10 client.4616.objecter ms_dispatch 0x7fdd60000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:30:55.330 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.336+0000 7fdd5d7fa640 3 client.4616.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:30:55.330 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.336+0000 7fdd5d7fa640 3 client.4616.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:30:55.330 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.336+0000 7fdd66d5e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:55.331 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.336+0000 7fdd66d5e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:30:55.331 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.336+0000 7fdd5d7fa640 20 client.4616.objecter dump_active .. 0 homeless 2024-03-29T11:30:55.331 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.336+0000 7fdd66d5e640 1 -- 172.21.15.114:0/2976632648 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd44005080 con 0x7fdd601422b0 2024-03-29T11:30:55.331 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.336+0000 7fdd64afa640 10 monclient: get_auth_request con 0x7fdd4006dbf0 auth_method 0 2024-03-29T11:30:55.331 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.336+0000 7fdd5d7fa640 10 client.4616.objecter ms_handle_connect 0x7fdd4006dbf0 2024-03-29T11:30:55.339 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.345+0000 7fdd5d7fa640 1 -- 172.21.15.114:0/2976632648 <== mon.2 v1:172.21.15.114:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdd50050260 con 0x7fdd601422b0 2024-03-29T11:30:55.339 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.345+0000 7fdd5d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:55.339 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.345+0000 7fdd5d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:30:55.667 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.673+0000 7fdd66d5e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:55.667 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.673+0000 7fdd66d5e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:55.667 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.673+0000 7fdd66d5e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:30:55.667 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.673+0000 7fdd66d5e640 1 -- 172.21.15.114:0/2976632648 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdd44005330 con 0x7fdd601422b0 2024-03-29T11:30:55.667 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.674+0000 7fdd5d7fa640 1 -- 172.21.15.114:0/2976632648 <== mon.2 v1:172.21.15.114:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3515021578) 0x7fdd50052360 con 0x7fdd601422b0 2024-03-29T11:30:55.668 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.674+0000 7fdd5d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:55.668 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.674+0000 7fdd5d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:30:55.670 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T11:30:55.670 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":11,"fsid":"b0de2b9b-c45a-44c7-b6ee-e83b39a1e108","created":"2024-03-29T11:20:49.420660+0000","modified":"2024-03-29T11:30:54.342208+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:30:54.342208+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a64785d3-6b11-499a-b706-cc827ab96a74","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"5882dadb-e7a8-4604-ba9a-418013c106c8","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"d7e8c409-bc66-4e4a-bbdb-420bad31388e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"a0682989-3595-4cd6-90a5-ab9111c278e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"67068120-e3ff-4817-bd06-44feb7479a35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09e8b-8a85-4997-bdea-75d754e6ee67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2f0d9ead-f47f-4cb3-92a4-0a5ad23cd831","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fdb6a25f-6bd8-4c65-9b9b-b460ba7673a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:30:55.672 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.678+0000 7fdd66d5e640 10 librados: watch_flush enter 2024-03-29T11:30:55.672 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.678+0000 7fdd66d5e640 10 librados: watch_flush exit 2024-03-29T11:30:55.672 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.678+0000 7fdd66d5e640 20 client.4616.objecter shutdown clearing up homeless session... 2024-03-29T11:30:55.672 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.678+0000 7fdd66d5e640 10 client.4616.objecter successfully canceled tick 2024-03-29T11:30:55.672 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.678+0000 7fdd66d5e640 1 -- 172.21.15.114:0/2976632648 >> v1:172.21.15.114:6816/4038755055 conn(0x7fdd4006dbf0 legacy=0x7fdd400700b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:55.672 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.678+0000 7fdd66d5e640 10 monclient: shutdown 2024-03-29T11:30:55.673 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.678+0000 7fdd66d5e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:55.673 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.678+0000 7fdd66d5e640 1 -- 172.21.15.114:0/2976632648 >> v1:172.21.15.114:6790/0 conn(0x7fdd601422b0 legacy=0x7fdd6012fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:55.673 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.679+0000 7fdd66d5e640 1 -- 172.21.15.114:0/2976632648 shutdown_connections 2024-03-29T11:30:55.673 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.679+0000 7fdd66d5e640 1 -- 172.21.15.114:0/2976632648 >> 172.21.15.114:0/2976632648 conn(0x7fdd60066a00 msgr2=0x7fdd600b3d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:55.673 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.679+0000 7fdd66d5e640 1 -- 172.21.15.114:0/2976632648 shutdown_connections 2024-03-29T11:30:55.673 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.679+0000 7fdd66d5e640 1 -- 172.21.15.114:0/2976632648 wait complete. 2024-03-29T11:30:55.673 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:30:55.679+0000 7fdd66d5e640 1 librados: shutdown 2024-03-29T11:30:55.689 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:31:02.496 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:31:02.645 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.651+0000 7fe0739f6640 10 monclient: get_monmap_and_config 2024-03-29T11:31:02.645 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.651+0000 7fe0739f6640 10 monclient: build_initial_monmap 2024-03-29T11:31:02.645 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.651+0000 7fe0739f6640 10 monclient: monmap: 2024-03-29T11:31:02.646 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:31:02.646 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:31:02.646 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:31:02.652801+0000 2024-03-29T11:31:02.646 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:31:02.652801+0000 2024-03-29T11:31:02.646 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:31:02.646 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:31:02.646 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:31:02.646 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T11:31:02.646 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T11:31:02.646 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:31:02.650 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.655+0000 7fe0739f6640 1 Processor -- start 2024-03-29T11:31:02.650 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.655+0000 7fe0739f6640 1 -- start start 2024-03-29T11:31:02.650 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.655+0000 7fe0739f6640 10 monclient: init 2024-03-29T11:31:02.650 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.656+0000 7fe0739f6640 10 monclient: _reopen_session rank -1 2024-03-29T11:31:02.650 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.656+0000 7fe0739f6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:31:02.650 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.656+0000 7fe0739f6640 10 monclient(hunting): picked mon.noname-b con 0x7fe064006e30 addr v1:172.21.15.114:6790/0 2024-03-29T11:31:02.650 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.656+0000 7fe0739f6640 10 monclient(hunting): picked mon.noname-a con 0x7fe064035e00 addr v1:172.21.15.114:6789/0 2024-03-29T11:31:02.650 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.656+0000 7fe0739f6640 10 monclient(hunting): picked mon.noname-c con 0x7fe0640c9670 addr v1:172.21.15.176:6789/0 2024-03-29T11:31:02.650 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.656+0000 7fe0739f6640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe064035970 con 0x7fe064035e00 2024-03-29T11:31:02.650 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.656+0000 7fe0739f6640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe064036280 con 0x7fe0640c9670 2024-03-29T11:31:02.650 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.656+0000 7fe0739f6640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe064035640 con 0x7fe064006e30 2024-03-29T11:31:02.650 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.656+0000 7fe0739f6640 10 monclient(hunting): _renew_subs 2024-03-29T11:31:02.650 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.656+0000 7fe0739f6640 10 monclient(hunting): authenticate will time out at 1532.838432s 2024-03-29T11:31:02.653 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.660+0000 7fe071792640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7fe064006e30 0x7fe0640cbcc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:52151/0 (socket says 172.21.15.114:52151) 2024-03-29T11:31:02.653 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.660+0000 7fe071792640 1 -- 172.21.15.114:0/786936750 learned_addr learned my addr 172.21.15.114:0/786936750 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:31:02.655 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.661+0000 7fe063fff640 1 -- 172.21.15.114:0/786936750 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3927583548 0 0) 0x7fe064035640 con 0x7fe064006e30 2024-03-29T11:31:02.655 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.661+0000 7fe063fff640 10 monclient(hunting): _negotiate 2024-03-29T11:31:02.655 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.661+0000 7fe063fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:02.656 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.661+0000 7fe063fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:02.656 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.661+0000 7fe063fff640 10 monclient(hunting): my global_id is 4619 2024-03-29T11:31:02.656 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.661+0000 7fe063fff640 1 -- 172.21.15.114:0/786936750 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe058003520 con 0x7fe064006e30 2024-03-29T11:31:02.659 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.662+0000 7fe063fff640 1 -- 172.21.15.114:0/786936750 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 181447760 0 0) 0x7fe064036280 con 0x7fe0640c9670 2024-03-29T11:31:02.659 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.662+0000 7fe063fff640 10 monclient(hunting): _negotiate 2024-03-29T11:31:02.659 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.662+0000 7fe063fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:02.659 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.662+0000 7fe063fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:02.659 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.662+0000 7fe063fff640 10 monclient(hunting): my global_id is 4555 2024-03-29T11:31:02.659 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.662+0000 7fe063fff640 1 -- 172.21.15.114:0/786936750 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe064035640 con 0x7fe0640c9670 2024-03-29T11:31:02.659 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.662+0000 7fe063fff640 1 -- 172.21.15.114:0/786936750 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 166613572 0 0) 0x7fe058003520 con 0x7fe064006e30 2024-03-29T11:31:02.660 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.662+0000 7fe063fff640 1 -- 172.21.15.114:0/786936750 >> v1:172.21.15.176:6789/0 conn(0x7fe0640c9670 legacy=0x7fe0640d7560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:02.660 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.662+0000 7fe063fff640 1 -- 172.21.15.114:0/786936750 >> v1:172.21.15.114:6789/0 conn(0x7fe064035e00 legacy=0x7fe0640c8f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:02.660 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.662+0000 7fe063fff640 10 monclient: _finish_hunting 0 2024-03-29T11:31:02.660 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.662+0000 7fe063fff640 1 monclient: found mon.noname-b 2024-03-29T11:31:02.660 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.662+0000 7fe063fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:31:02.660 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.662+0000 7fe063fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.114:6790/0 2024-03-29T11:31:02.660 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.662+0000 7fe063fff640 1 -- 172.21.15.114:0/786936750 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe064034e20 con 0x7fe064006e30 2024-03-29T11:31:02.660 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.662+0000 7fe063fff640 10 monclient: _finish_auth 0 2024-03-29T11:31:02.660 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.662+0000 7fe063fff640 10 monclient: _check_auth_tickets 2024-03-29T11:31:02.660 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.662+0000 7fe063fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:31:02.660 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.662+0000 7fe063fff640 1 -- 172.21.15.114:0/786936750 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fe06c04f730 con 0x7fe064006e30 2024-03-29T11:31:02.660 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.662+0000 7fe063fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:02.660 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.662+0000 7fe063fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:31:02.660 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.662+0000 7fe063fff640 10 monclient: dump: 2024-03-29T11:31:02.660 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:31:02.661 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:31:02.661 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:31:02.661 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:31:02.661 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:02.661 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:31:02.661 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:31:02.661 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:31:02.661 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:31:02.661 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:31:02.661 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.663+0000 7fe0739f6640 5 monclient: authenticate success, global_id 4619 2024-03-29T11:31:02.661 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.663+0000 7fe0739f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:31:02.661 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.664+0000 7fe063fff640 1 -- 172.21.15.114:0/786936750 <== mon.2 v1:172.21.15.114:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe06c075f10 con 0x7fe064006e30 2024-03-29T11:31:02.661 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.664+0000 7fe063fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:31:02.661 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.664+0000 7fe063fff640 1 -- 172.21.15.114:0/786936750 <== mon.2 v1:172.21.15.114:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fe06c0758b0 con 0x7fe064006e30 2024-03-29T11:31:02.662 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.664+0000 7fe063fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:02.662 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.664+0000 7fe063fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:31:02.662 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.664+0000 7fe063fff640 10 monclient: dump: 2024-03-29T11:31:02.663 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:31:02.663 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:31:02.663 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:31:02.663 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:31:02.663 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:02.663 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:31:02.663 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:31:02.663 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:31:02.663 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:31:02.663 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:31:02.663 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.664+0000 7fe0739f6640 10 monclient: get_monmap_and_config success 2024-03-29T11:31:02.663 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.664+0000 7fe0739f6640 4 set_mon_vals no callback set 2024-03-29T11:31:02.663 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.665+0000 7fe0739f6640 10 monclient: shutdown 2024-03-29T11:31:02.663 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.665+0000 7fe0739f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:31:02.664 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.665+0000 7fe0739f6640 1 -- 172.21.15.114:0/786936750 >> v1:172.21.15.114:6790/0 conn(0x7fe064006e30 legacy=0x7fe0640cbcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:02.664 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.665+0000 7fe0739f6640 1 -- 172.21.15.114:0/786936750 shutdown_connections 2024-03-29T11:31:02.664 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.665+0000 7fe0739f6640 1 -- 172.21.15.114:0/786936750 >> 172.21.15.114:0/786936750 conn(0x7fe064005c00 msgr2=0x7fe064006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:02.666 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.667+0000 7fe0739f6640 1 -- 172.21.15.114:0/786936750 shutdown_connections 2024-03-29T11:31:02.666 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.667+0000 7fe0739f6640 1 -- 172.21.15.114:0/786936750 wait complete. 2024-03-29T11:31:02.666 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.668+0000 7fe0739f6640 10 monclient: build_initial_monmap 2024-03-29T11:31:02.666 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.668+0000 7fe0739f6640 10 monclient: monmap: 2024-03-29T11:31:02.666 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:31:02.666 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:31:02.666 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T11:31:02.666 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T11:31:02.666 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:31:02.666 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:31:02.667 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:31:02.667 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T11:31:02.667 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T11:31:02.667 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:31:02.667 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.668+0000 7fe0739f6640 1 librados: starting msgr at 2024-03-29T11:31:02.667 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.668+0000 7fe0739f6640 1 librados: starting objecter 2024-03-29T11:31:02.667 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.668+0000 7fe0739f6640 1 Processor -- start 2024-03-29T11:31:02.667 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.669+0000 7fe0739f6640 1 -- start start 2024-03-29T11:31:02.667 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.669+0000 7fe0739f6640 1 librados: setting wanted keys 2024-03-29T11:31:02.667 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.669+0000 7fe0739f6640 1 librados: calling monclient init 2024-03-29T11:31:02.667 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.669+0000 7fe0739f6640 10 monclient: init 2024-03-29T11:31:02.667 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.670+0000 7fe0739f6640 10 monclient: _reopen_session rank -1 2024-03-29T11:31:02.667 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.670+0000 7fe0739f6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:31:02.667 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.670+0000 7fe0739f6640 10 monclient(hunting): picked mon.noname-a con 0x7fe064035e00 addr v1:172.21.15.114:6789/0 2024-03-29T11:31:02.667 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.670+0000 7fe0739f6640 10 monclient(hunting): picked mon.noname-b con 0x7fe0640c9670 addr v1:172.21.15.176:6789/0 2024-03-29T11:31:02.668 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.670+0000 7fe0739f6640 10 monclient(hunting): picked mon.noname-c con 0x7fe0641648a0 addr v1:172.21.15.114:6790/0 2024-03-29T11:31:02.668 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.670+0000 7fe0739f6640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe064033e50 con 0x7fe064035e00 2024-03-29T11:31:02.668 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.670+0000 7fe0739f6640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe064033ad0 con 0x7fe0640c9670 2024-03-29T11:31:02.668 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.670+0000 7fe0739f6640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe064032a50 con 0x7fe0641648a0 2024-03-29T11:31:02.668 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.670+0000 7fe0739f6640 10 monclient(hunting): _renew_subs 2024-03-29T11:31:02.668 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.670+0000 7fe0739f6640 10 monclient(hunting): authenticate will time out at 1532.852402s 2024-03-29T11:31:02.671 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.677+0000 7fe071f93640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7fe0641648a0 0x7fe0640c0920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:52231/0 (socket says 172.21.15.114:52231) 2024-03-29T11:31:02.671 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.677+0000 7fe071f93640 1 -- 172.21.15.114:0/4240870242 learned_addr learned my addr 172.21.15.114:0/4240870242 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:31:02.671 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.678+0000 7fe061ffb640 10 client.?.objecter ms_handle_connect 0x7fe0641648a0 2024-03-29T11:31:02.671 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.678+0000 7fe061ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:31:02.675 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.678+0000 7fe061ffb640 1 -- 172.21.15.114:0/4240870242 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3248637683 0 0) 0x7fe064032a50 con 0x7fe0641648a0 2024-03-29T11:31:02.675 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.678+0000 7fe061ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:31:02.675 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.678+0000 7fe061ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:02.675 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.678+0000 7fe061ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:02.675 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.678+0000 7fe061ffb640 10 monclient(hunting): my global_id is 4622 2024-03-29T11:31:02.675 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.678+0000 7fe061ffb640 1 -- 172.21.15.114:0/4240870242 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe050004eb0 con 0x7fe0641648a0 2024-03-29T11:31:02.675 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.678+0000 7fe061ffb640 1 -- 172.21.15.114:0/4240870242 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 819777078 0 0) 0x7fe050004eb0 con 0x7fe0641648a0 2024-03-29T11:31:02.675 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.678+0000 7fe061ffb640 1 -- 172.21.15.114:0/4240870242 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe064032a50 con 0x7fe0641648a0 2024-03-29T11:31:02.675 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.678+0000 7fe061ffb640 1 -- 172.21.15.114:0/4240870242 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fe05c001890 con 0x7fe0641648a0 2024-03-29T11:31:02.675 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.678+0000 7fe061ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:02.675 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.678+0000 7fe061ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:31:02.675 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.678+0000 7fe061ffb640 10 monclient(hunting): dump: 2024-03-29T11:31:02.675 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:31:02.676 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:31:02.676 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:31:02.676 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:31:02.676 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:02.676 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:31:02.676 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:31:02.676 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:31:02.676 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:31:02.676 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:31:02.676 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.678+0000 7fe061ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:31:02.676 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.679+0000 7fe061ffb640 10 client.?.objecter ms_handle_connect 0x7fe0640c9670 2024-03-29T11:31:02.676 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.679+0000 7fe061ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:31:02.676 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.679+0000 7fe061ffb640 1 -- 172.21.15.114:0/4240870242 <== mon.2 v1:172.21.15.114:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1541166396 0 0) 0x7fe064032a50 con 0x7fe0641648a0 2024-03-29T11:31:02.676 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.679+0000 7fe061ffb640 1 -- 172.21.15.114:0/4240870242 >> v1:172.21.15.176:6789/0 conn(0x7fe0640c9670 legacy=0x7fe064164190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:02.677 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.679+0000 7fe061ffb640 1 -- 172.21.15.114:0/4240870242 >> v1:172.21.15.114:6789/0 conn(0x7fe064035e00 legacy=0x7fe064032d50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:31:02.677 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.679+0000 7fe061ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:31:02.677 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.679+0000 7fe061ffb640 1 monclient: found mon.c 2024-03-29T11:31:02.677 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.679+0000 7fe061ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:31:02.677 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.680+0000 7fe061ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:31:02.677 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.680+0000 7fe061ffb640 1 -- 172.21.15.114:0/4240870242 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0640cad40 con 0x7fe0641648a0 2024-03-29T11:31:02.677 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.680+0000 7fe061ffb640 10 monclient: _finish_auth 0 2024-03-29T11:31:02.677 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.680+0000 7fe061ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:31:02.677 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.680+0000 7fe061ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:31:02.677 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.680+0000 7fe0739f6640 5 monclient: authenticate success, global_id 4622 2024-03-29T11:31:02.677 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.680+0000 7fe0739f6640 10 monclient: _renew_subs 2024-03-29T11:31:02.677 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.680+0000 7fe0739f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:31:02.677 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.680+0000 7fe0739f6640 1 -- 172.21.15.114:0/4240870242 --> v1:172.21.15.114:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0640caf90 con 0x7fe0641648a0 2024-03-29T11:31:02.677 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.680+0000 7fe0739f6640 10 client.4622.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:31:02.677 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.680+0000 7fe0739f6640 10 monclient: _renew_subs 2024-03-29T11:31:02.678 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.680+0000 7fe0739f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:31:02.678 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.680+0000 7fe0739f6640 1 -- 172.21.15.114:0/4240870242 --> v1:172.21.15.114:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0640c1210 con 0x7fe0641648a0 2024-03-29T11:31:02.678 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.680+0000 7fe0739f6640 1 librados: init done 2024-03-29T11:31:02.680 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.683+0000 7fe0739f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:02.680 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.683+0000 7fe061ffb640 1 -- 172.21.15.114:0/4240870242 <== mon.2 v1:172.21.15.114:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe05c0019f0 con 0x7fe0641648a0 2024-03-29T11:31:02.680 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.683+0000 7fe061ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:31:02.680 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.683+0000 7fe061ffb640 1 -- 172.21.15.114:0/4240870242 <== mon.2 v1:172.21.15.114:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fe05c0038b0 con 0x7fe0641648a0 2024-03-29T11:31:02.680 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.683+0000 7fe061ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:02.680 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.683+0000 7fe061ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:31:02.680 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.683+0000 7fe061ffb640 10 monclient: dump: 2024-03-29T11:31:02.680 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:31:02.680 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:31:02.680 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:31:02.680 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:31:02.681 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:02.681 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:31:02.681 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:31:02.681 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:31:02.681 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:31:02.681 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:31:02.681 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.683+0000 7fe0637fe640 4 set_mon_vals no callback set 2024-03-29T11:31:02.681 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.685+0000 7fe061ffb640 1 -- 172.21.15.114:0/4240870242 <== mon.2 v1:172.21.15.114:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1594292461 0 0) 0x7fe05c003ad0 con 0x7fe0641648a0 2024-03-29T11:31:02.681 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.685+0000 7fe0739f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:02.681 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.685+0000 7fe0739f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:31:02.681 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.685+0000 7fe0739f6640 1 -- 172.21.15.114:0/4240870242 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe064007b70 con 0x7fe0641648a0 2024-03-29T11:31:02.681 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.686+0000 7fe061ffb640 1 -- 172.21.15.114:0/4240870242 <== mon.2 v1:172.21.15.114:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1864569634 0 0) 0x7fe05c084a80 con 0x7fe0641648a0 2024-03-29T11:31:02.681 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.686+0000 7fe071792640 10 monclient: get_auth_request con 0x7fe05006daf0 auth_method 0 2024-03-29T11:31:02.681 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.686+0000 7fe061ffb640 10 client.4622.objecter ms_dispatch 0x7fe06c000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:31:02.681 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.686+0000 7fe061ffb640 3 client.4622.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:31:02.682 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.686+0000 7fe061ffb640 3 client.4622.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:31:02.682 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.686+0000 7fe061ffb640 20 client.4622.objecter dump_active .. 0 homeless 2024-03-29T11:31:02.682 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.687+0000 7fe061ffb640 10 client.4622.objecter ms_handle_connect 0x7fe05006daf0 2024-03-29T11:31:02.690 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.696+0000 7fe061ffb640 1 -- 172.21.15.114:0/4240870242 <== mon.2 v1:172.21.15.114:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe05c050360 con 0x7fe0641648a0 2024-03-29T11:31:02.690 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.696+0000 7fe061ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:02.690 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:02.697+0000 7fe061ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:31:03.017 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:03.023+0000 7fe0739f6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:31:03.017 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:03.023+0000 7fe0739f6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:31:03.017 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:03.023+0000 7fe0739f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:31:03.017 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:03.023+0000 7fe0739f6640 1 -- 172.21.15.114:0/4240870242 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe0640cc380 con 0x7fe0641648a0 2024-03-29T11:31:03.017 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:03.023+0000 7fe061ffb640 1 -- 172.21.15.114:0/4240870242 <== mon.2 v1:172.21.15.114:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3515021578) 0x7fe05c052290 con 0x7fe0641648a0 2024-03-29T11:31:03.017 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:03.023+0000 7fe061ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:31:03.017 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:03.023+0000 7fe061ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:31:03.019 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T11:31:03.019 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":11,"fsid":"b0de2b9b-c45a-44c7-b6ee-e83b39a1e108","created":"2024-03-29T11:20:49.420660+0000","modified":"2024-03-29T11:30:54.342208+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:30:54.342208+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a64785d3-6b11-499a-b706-cc827ab96a74","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"5882dadb-e7a8-4604-ba9a-418013c106c8","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"d7e8c409-bc66-4e4a-bbdb-420bad31388e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"a0682989-3595-4cd6-90a5-ab9111c278e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"67068120-e3ff-4817-bd06-44feb7479a35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09e8b-8a85-4997-bdea-75d754e6ee67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2f0d9ead-f47f-4cb3-92a4-0a5ad23cd831","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fdb6a25f-6bd8-4c65-9b9b-b460ba7673a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:31:03.021 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:03.027+0000 7fe0739f6640 10 librados: watch_flush enter 2024-03-29T11:31:03.021 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:03.027+0000 7fe0739f6640 10 librados: watch_flush exit 2024-03-29T11:31:03.021 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:03.027+0000 7fe0739f6640 20 client.4622.objecter shutdown clearing up homeless session... 2024-03-29T11:31:03.021 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:03.027+0000 7fe0739f6640 10 client.4622.objecter successfully canceled tick 2024-03-29T11:31:03.021 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:03.027+0000 7fe0739f6640 1 -- 172.21.15.114:0/4240870242 >> v1:172.21.15.114:6816/4038755055 conn(0x7fe05006daf0 legacy=0x7fe05006ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:03.021 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:03.027+0000 7fe0739f6640 10 monclient: shutdown 2024-03-29T11:31:03.021 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:03.028+0000 7fe0739f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:31:03.022 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:03.028+0000 7fe0739f6640 1 -- 172.21.15.114:0/4240870242 >> v1:172.21.15.114:6790/0 conn(0x7fe0641648a0 legacy=0x7fe0640c0920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:03.022 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:03.028+0000 7fe0739f6640 1 -- 172.21.15.114:0/4240870242 shutdown_connections 2024-03-29T11:31:03.022 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:03.028+0000 7fe0739f6640 1 -- 172.21.15.114:0/4240870242 >> 172.21.15.114:0/4240870242 conn(0x7fe064005c00 msgr2=0x7fe064005fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:03.025 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:03.032+0000 7fe0739f6640 1 -- 172.21.15.114:0/4240870242 shutdown_connections 2024-03-29T11:31:03.026 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:03.032+0000 7fe0739f6640 1 -- 172.21.15.114:0/4240870242 wait complete. 2024-03-29T11:31:03.026 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:03.032+0000 7fe0739f6640 1 librados: shutdown 2024-03-29T11:31:03.042 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:31:09.849 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:31:10.001 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.007+0000 7f44e78c8640 10 monclient: get_monmap_and_config 2024-03-29T11:31:10.001 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.007+0000 7f44e78c8640 10 monclient: build_initial_monmap 2024-03-29T11:31:10.001 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.007+0000 7f44e78c8640 10 monclient: monmap: 2024-03-29T11:31:10.001 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:31:10.001 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:31:10.001 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:31:10.008642+0000 2024-03-29T11:31:10.001 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:31:10.008642+0000 2024-03-29T11:31:10.001 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:31:10.001 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:31:10.001 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:31:10.002 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T11:31:10.002 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T11:31:10.002 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:31:10.003 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.009+0000 7f44e78c8640 1 Processor -- start 2024-03-29T11:31:10.003 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.009+0000 7f44e78c8640 1 -- start start 2024-03-29T11:31:10.003 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.009+0000 7f44e78c8640 10 monclient: init 2024-03-29T11:31:10.004 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.010+0000 7f44e78c8640 10 monclient: _reopen_session rank -1 2024-03-29T11:31:10.004 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.010+0000 7f44e78c8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:31:10.004 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.010+0000 7f44e78c8640 10 monclient(hunting): picked mon.noname-b con 0x7f44e01422b0 addr v1:172.21.15.114:6790/0 2024-03-29T11:31:10.004 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.010+0000 7f44e78c8640 10 monclient(hunting): picked mon.noname-c con 0x7f44e0142690 addr v1:172.21.15.176:6789/0 2024-03-29T11:31:10.005 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.010+0000 7f44e78c8640 10 monclient(hunting): picked mon.noname-a con 0x7f44e00b17d0 addr v1:172.21.15.114:6789/0 2024-03-29T11:31:10.005 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.010+0000 7f44e78c8640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44e0142c80 con 0x7f44e00b17d0 2024-03-29T11:31:10.005 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.010+0000 7f44e78c8640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44e00aae40 con 0x7f44e0142690 2024-03-29T11:31:10.005 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.010+0000 7f44e78c8640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44e013fbd0 con 0x7f44e01422b0 2024-03-29T11:31:10.005 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.010+0000 7f44e78c8640 10 monclient(hunting): _renew_subs 2024-03-29T11:31:10.005 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.010+0000 7f44e78c8640 10 monclient(hunting): authenticate will time out at 1540.192906s 2024-03-29T11:31:10.006 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.011+0000 7f44e5664640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f44e01422b0 0x7f44e013f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:49326/0 (socket says 172.21.15.114:49326) 2024-03-29T11:31:10.006 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.011+0000 7f44e5664640 1 -- 172.21.15.114:0/2745121268 learned_addr learned my addr 172.21.15.114:0/2745121268 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:31:10.006 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.012+0000 7f44d7fff640 1 -- 172.21.15.114:0/2745121268 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 135481845 0 0) 0x7f44e013fbd0 con 0x7f44e01422b0 2024-03-29T11:31:10.006 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.012+0000 7f44d7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:31:10.006 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.012+0000 7f44d7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:10.007 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.012+0000 7f44d7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:10.007 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.012+0000 7f44d7fff640 10 monclient(hunting): my global_id is 4625 2024-03-29T11:31:10.007 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.012+0000 7f44d7fff640 1 -- 172.21.15.114:0/2745121268 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44cc003520 con 0x7f44e01422b0 2024-03-29T11:31:10.007 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.012+0000 7f44d7fff640 1 -- 172.21.15.114:0/2745121268 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1211067267 0 0) 0x7f44cc003520 con 0x7f44e01422b0 2024-03-29T11:31:10.007 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.012+0000 7f44d7fff640 1 -- 172.21.15.114:0/2745121268 >> v1:172.21.15.176:6789/0 conn(0x7f44e0142690 legacy=0x7f44e00b2590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:31:10.009 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.014+0000 7f44d7fff640 1 -- 172.21.15.114:0/2745121268 >> v1:172.21.15.114:6789/0 conn(0x7f44e00b17d0 legacy=0x7f44e00b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:10.009 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.014+0000 7f44d7fff640 10 monclient: _finish_hunting 0 2024-03-29T11:31:10.009 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.014+0000 7f44d7fff640 1 monclient: found mon.noname-b 2024-03-29T11:31:10.009 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.014+0000 7f44d7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:31:10.009 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.014+0000 7f44d7fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.114:6790/0 2024-03-29T11:31:10.009 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.014+0000 7f44d7fff640 1 -- 172.21.15.114:0/2745121268 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44e0151940 con 0x7f44e01422b0 2024-03-29T11:31:10.010 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.014+0000 7f44d7fff640 10 monclient: _finish_auth 0 2024-03-29T11:31:10.010 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.014+0000 7f44d7fff640 10 monclient: _check_auth_tickets 2024-03-29T11:31:10.010 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.014+0000 7f44d7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:31:10.010 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.014+0000 7f44d7fff640 1 -- 172.21.15.114:0/2745121268 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f44d0001690 con 0x7f44e01422b0 2024-03-29T11:31:10.010 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.014+0000 7f44e78c8640 5 monclient: authenticate success, global_id 4625 2024-03-29T11:31:10.010 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.014+0000 7f44e78c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:31:10.010 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.014+0000 7f44d7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:10.010 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.014+0000 7f44d7fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:31:10.010 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.014+0000 7f44d7fff640 10 monclient: dump: 2024-03-29T11:31:10.010 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:31:10.010 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:31:10.010 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:31:10.010 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:31:10.010 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:10.011 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:31:10.011 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:31:10.011 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:31:10.011 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:31:10.011 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:31:10.011 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.014+0000 7f44e78c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:31:10.011 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.017+0000 7f44d7fff640 1 -- 172.21.15.114:0/2745121268 <== mon.2 v1:172.21.15.114:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f44d00017f0 con 0x7f44e01422b0 2024-03-29T11:31:10.011 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.017+0000 7f44d7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:31:10.012 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.018+0000 7f44e78c8640 10 monclient: get_monmap_and_config success 2024-03-29T11:31:10.013 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.018+0000 7f44e78c8640 4 set_mon_vals no callback set 2024-03-29T11:31:10.013 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.018+0000 7f44e78c8640 10 monclient: shutdown 2024-03-29T11:31:10.013 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.018+0000 7f44e78c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:31:10.013 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.018+0000 7f44e78c8640 1 -- 172.21.15.114:0/2745121268 >> v1:172.21.15.114:6790/0 conn(0x7f44e01422b0 legacy=0x7f44e013f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:10.013 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.019+0000 7f44e78c8640 1 -- 172.21.15.114:0/2745121268 shutdown_connections 2024-03-29T11:31:10.013 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.019+0000 7f44e78c8640 1 -- 172.21.15.114:0/2745121268 >> 172.21.15.114:0/2745121268 conn(0x7f44e0066a00 msgr2=0x7f44e0066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:10.015 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.022+0000 7f44e78c8640 1 -- 172.21.15.114:0/2745121268 shutdown_connections 2024-03-29T11:31:10.015 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.022+0000 7f44e78c8640 1 -- 172.21.15.114:0/2745121268 wait complete. 2024-03-29T11:31:10.016 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.023+0000 7f44e78c8640 10 monclient: build_initial_monmap 2024-03-29T11:31:10.017 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.023+0000 7f44e78c8640 10 monclient: monmap: 2024-03-29T11:31:10.017 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:31:10.017 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:31:10.017 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T11:31:10.017 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T11:31:10.017 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:31:10.017 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:31:10.017 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:31:10.017 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T11:31:10.017 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T11:31:10.017 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:31:10.017 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.023+0000 7f44e78c8640 1 librados: starting msgr at 2024-03-29T11:31:10.017 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.023+0000 7f44e78c8640 1 librados: starting objecter 2024-03-29T11:31:10.017 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.023+0000 7f44e78c8640 1 Processor -- start 2024-03-29T11:31:10.018 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.023+0000 7f44e78c8640 1 -- start start 2024-03-29T11:31:10.018 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.023+0000 7f44e78c8640 1 librados: setting wanted keys 2024-03-29T11:31:10.018 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.023+0000 7f44e78c8640 1 librados: calling monclient init 2024-03-29T11:31:10.018 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.023+0000 7f44e78c8640 10 monclient: init 2024-03-29T11:31:10.018 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.024+0000 7f44e78c8640 10 monclient: _reopen_session rank -1 2024-03-29T11:31:10.018 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.024+0000 7f44e78c8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:31:10.019 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.024+0000 7f44e78c8640 10 monclient(hunting): picked mon.noname-a con 0x7f44e00b17d0 addr v1:172.21.15.114:6789/0 2024-03-29T11:31:10.027 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.026+0000 7f44e78c8640 10 monclient(hunting): picked mon.noname-b con 0x7f44e01422b0 addr v1:172.21.15.176:6789/0 2024-03-29T11:31:10.027 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.026+0000 7f44e4e63640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f44e01422b0 0x7f44e0130610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.114:55609/0 (socket says 172.21.15.114:55609) 2024-03-29T11:31:10.027 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.027+0000 7f44e4e63640 1 -- 172.21.15.114:0/3517462735 learned_addr learned my addr 172.21.15.114:0/3517462735 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:31:10.027 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.027+0000 7f44e78c8640 10 monclient(hunting): picked mon.noname-c con 0x7f44e0142690 addr v1:172.21.15.114:6790/0 2024-03-29T11:31:10.027 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.027+0000 7f44e78c8640 1 -- 172.21.15.114:0/3517462735 --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44e00ac610 con 0x7f44e00b17d0 2024-03-29T11:31:10.028 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.027+0000 7f44e78c8640 1 -- 172.21.15.114:0/3517462735 --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44e00ac8f0 con 0x7f44e01422b0 2024-03-29T11:31:10.028 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.027+0000 7f44e78c8640 1 -- 172.21.15.114:0/3517462735 --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44e00acbd0 con 0x7f44e0142690 2024-03-29T11:31:10.028 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.027+0000 7f44e78c8640 10 monclient(hunting): _renew_subs 2024-03-29T11:31:10.028 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.027+0000 7f44e78c8640 10 monclient(hunting): authenticate will time out at 1540.209263s 2024-03-29T11:31:10.028 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.027+0000 7f44d5ffb640 10 client.?.objecter ms_handle_connect 0x7f44e0142690 2024-03-29T11:31:10.028 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.027+0000 7f44d5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:31:10.028 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.027+0000 7f44d5ffb640 1 -- 172.21.15.114:0/3517462735 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3276739475 0 0) 0x7f44e00acbd0 con 0x7f44e0142690 2024-03-29T11:31:10.028 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.027+0000 7f44d5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:31:10.028 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.027+0000 7f44d5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:10.028 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.027+0000 7f44d5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:10.028 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.027+0000 7f44d5ffb640 10 monclient(hunting): my global_id is 4628 2024-03-29T11:31:10.028 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.027+0000 7f44d5ffb640 1 -- 172.21.15.114:0/3517462735 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44c4004eb0 con 0x7f44e0142690 2024-03-29T11:31:10.028 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.027+0000 7f44d5ffb640 10 client.?.objecter ms_handle_connect 0x7f44e00b17d0 2024-03-29T11:31:10.028 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.027+0000 7f44d5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:31:10.028 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.028+0000 7f44d5ffb640 10 client.?.objecter ms_handle_connect 0x7f44e01422b0 2024-03-29T11:31:10.028 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.028+0000 7f44d5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:31:10.029 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.028+0000 7f44d5ffb640 1 -- 172.21.15.114:0/3517462735 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1114766046 0 0) 0x7f44c4004eb0 con 0x7f44e0142690 2024-03-29T11:31:10.029 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.028+0000 7f44d5ffb640 1 -- 172.21.15.114:0/3517462735 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f44e00acbd0 con 0x7f44e0142690 2024-03-29T11:31:10.029 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.028+0000 7f44d5ffb640 1 -- 172.21.15.114:0/3517462735 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f44dc0017e0 con 0x7f44e0142690 2024-03-29T11:31:10.029 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.028+0000 7f44d5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:10.029 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.028+0000 7f44d5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:31:10.029 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.028+0000 7f44d5ffb640 10 monclient(hunting): dump: 2024-03-29T11:31:10.029 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:31:10.029 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:31:10.029 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:31:10.029 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:31:10.029 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:10.029 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:31:10.029 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:31:10.029 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:31:10.029 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:31:10.030 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:31:10.030 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.028+0000 7f44d5ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:31:10.030 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.028+0000 7f44d5ffb640 1 -- 172.21.15.114:0/3517462735 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3229980709 0 0) 0x7f44e00ac610 con 0x7f44e00b17d0 2024-03-29T11:31:10.030 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.028+0000 7f44d5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:31:10.030 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.028+0000 7f44d5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:10.030 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.028+0000 7f44d5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:10.030 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.028+0000 7f44d5ffb640 10 monclient(hunting): my global_id is 4578 2024-03-29T11:31:10.030 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.028+0000 7f44d5ffb640 1 -- 172.21.15.114:0/3517462735 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44c4004eb0 con 0x7f44e00b17d0 2024-03-29T11:31:10.030 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.028+0000 7f44d5ffb640 1 -- 172.21.15.114:0/3517462735 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3949611060 0 0) 0x7f44e00ac8f0 con 0x7f44e01422b0 2024-03-29T11:31:10.030 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.029+0000 7f44d5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:31:10.030 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.029+0000 7f44d5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:10.030 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.029+0000 7f44d5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:10.030 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.029+0000 7f44d5ffb640 10 monclient(hunting): my global_id is 4558 2024-03-29T11:31:10.030 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.029+0000 7f44d5ffb640 1 -- 172.21.15.114:0/3517462735 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44e00ac610 con 0x7f44e01422b0 2024-03-29T11:31:10.030 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.029+0000 7f44d5ffb640 1 -- 172.21.15.114:0/3517462735 <== mon.2 v1:172.21.15.114:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2717944767 0 0) 0x7f44e00acbd0 con 0x7f44e0142690 2024-03-29T11:31:10.031 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.029+0000 7f44d5ffb640 1 -- 172.21.15.114:0/3517462735 >> v1:172.21.15.176:6789/0 conn(0x7f44e01422b0 legacy=0x7f44e0130610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:10.031 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.029+0000 7f44d5ffb640 1 -- 172.21.15.114:0/3517462735 >> v1:172.21.15.114:6789/0 conn(0x7f44e00b17d0 legacy=0x7f44e012ff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:10.031 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.029+0000 7f44d5ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:31:10.031 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.029+0000 7f44d5ffb640 1 monclient: found mon.c 2024-03-29T11:31:10.031 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.029+0000 7f44d5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:31:10.031 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.029+0000 7f44d5ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:31:10.031 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.029+0000 7f44d5ffb640 1 -- 172.21.15.114:0/3517462735 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44e0132b10 con 0x7f44e0142690 2024-03-29T11:31:10.031 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.029+0000 7f44d5ffb640 10 monclient: _finish_auth 0 2024-03-29T11:31:10.031 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.029+0000 7f44d5ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:31:10.031 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.029+0000 7f44d5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:31:10.031 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.030+0000 7f44e78c8640 5 monclient: authenticate success, global_id 4628 2024-03-29T11:31:10.031 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.030+0000 7f44e78c8640 10 monclient: _renew_subs 2024-03-29T11:31:10.031 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.030+0000 7f44e78c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:31:10.031 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.030+0000 7f44e78c8640 1 -- 172.21.15.114:0/3517462735 --> v1:172.21.15.114:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44e01325a0 con 0x7f44e0142690 2024-03-29T11:31:10.031 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.030+0000 7f44e78c8640 10 client.4628.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:31:10.031 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.030+0000 7f44e78c8640 10 monclient: _renew_subs 2024-03-29T11:31:10.032 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.030+0000 7f44e78c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:31:10.032 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.030+0000 7f44e78c8640 1 -- 172.21.15.114:0/3517462735 --> v1:172.21.15.114:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f44e00ab8e0 con 0x7f44e0142690 2024-03-29T11:31:10.032 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.030+0000 7f44e78c8640 1 librados: init done 2024-03-29T11:31:10.032 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.031+0000 7f44d5ffb640 1 -- 172.21.15.114:0/3517462735 <== mon.2 v1:172.21.15.114:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f44dc0019c0 con 0x7f44e0142690 2024-03-29T11:31:10.032 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.031+0000 7f44d5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:31:10.032 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.031+0000 7f44d5ffb640 1 -- 172.21.15.114:0/3517462735 <== mon.2 v1:172.21.15.114:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f44dc0037b0 con 0x7f44e0142690 2024-03-29T11:31:10.032 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.031+0000 7f44d5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:10.032 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.031+0000 7f44d77fe640 4 set_mon_vals no callback set 2024-03-29T11:31:10.032 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.031+0000 7f44d5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:31:10.032 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.031+0000 7f44d5ffb640 10 monclient: dump: 2024-03-29T11:31:10.032 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:31:10.032 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:31:10.032 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:31:10.032 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:31:10.033 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:10.033 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:31:10.033 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:31:10.033 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:31:10.033 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:31:10.033 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:31:10.033 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.032+0000 7f44e78c8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:10.033 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.032+0000 7f44d5ffb640 1 -- 172.21.15.114:0/3517462735 <== mon.2 v1:172.21.15.114:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1594292461 0 0) 0x7f44dc003910 con 0x7f44e0142690 2024-03-29T11:31:10.033 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.032+0000 7f44e78c8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:10.033 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.032+0000 7f44e78c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:31:10.033 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.032+0000 7f44e78c8640 1 -- 172.21.15.114:0/3517462735 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44c0005080 con 0x7f44e0142690 2024-03-29T11:31:10.033 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.033+0000 7f44e5664640 10 monclient: get_auth_request con 0x7f44c406dbe0 auth_method 0 2024-03-29T11:31:10.033 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.033+0000 7f44d5ffb640 1 -- 172.21.15.114:0/3517462735 <== mon.2 v1:172.21.15.114:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1864569634 0 0) 0x7f44dc083830 con 0x7f44e0142690 2024-03-29T11:31:10.034 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.033+0000 7f44d5ffb640 10 client.4628.objecter ms_dispatch 0x7f44e0000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:31:10.034 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.033+0000 7f44d5ffb640 3 client.4628.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:31:10.034 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.033+0000 7f44d5ffb640 3 client.4628.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:31:10.034 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.033+0000 7f44d5ffb640 20 client.4628.objecter dump_active .. 0 homeless 2024-03-29T11:31:10.034 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.033+0000 7f44d5ffb640 10 client.4628.objecter ms_handle_connect 0x7f44c406dbe0 2024-03-29T11:31:10.034 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.040+0000 7f44d5ffb640 1 -- 172.21.15.114:0/3517462735 <== mon.2 v1:172.21.15.114:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f44dc04ef80 con 0x7f44e0142690 2024-03-29T11:31:10.034 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.040+0000 7f44d5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:10.034 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.040+0000 7f44d5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:31:10.356 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.363+0000 7f44e78c8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:31:10.357 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.363+0000 7f44e78c8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:31:10.357 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.363+0000 7f44e78c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:31:10.357 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.363+0000 7f44e78c8640 1 -- 172.21.15.114:0/3517462735 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f44c0005330 con 0x7f44e0142690 2024-03-29T11:31:10.358 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.364+0000 7f44d5ffb640 1 -- 172.21.15.114:0/3517462735 <== mon.2 v1:172.21.15.114:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3515021578) 0x7f44dc051040 con 0x7f44e0142690 2024-03-29T11:31:10.358 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.364+0000 7f44d5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:31:10.358 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.364+0000 7f44d5ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:31:10.358 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T11:31:10.358 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":11,"fsid":"b0de2b9b-c45a-44c7-b6ee-e83b39a1e108","created":"2024-03-29T11:20:49.420660+0000","modified":"2024-03-29T11:30:54.342208+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:30:54.342208+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a64785d3-6b11-499a-b706-cc827ab96a74","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"5882dadb-e7a8-4604-ba9a-418013c106c8","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"d7e8c409-bc66-4e4a-bbdb-420bad31388e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"a0682989-3595-4cd6-90a5-ab9111c278e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"67068120-e3ff-4817-bd06-44feb7479a35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09e8b-8a85-4997-bdea-75d754e6ee67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2f0d9ead-f47f-4cb3-92a4-0a5ad23cd831","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fdb6a25f-6bd8-4c65-9b9b-b460ba7673a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:31:10.361 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.368+0000 7f44e78c8640 10 librados: watch_flush enter 2024-03-29T11:31:10.361 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.368+0000 7f44e78c8640 10 librados: watch_flush exit 2024-03-29T11:31:10.362 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.368+0000 7f44e78c8640 20 client.4628.objecter shutdown clearing up homeless session... 2024-03-29T11:31:10.362 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.368+0000 7f44e78c8640 10 client.4628.objecter successfully canceled tick 2024-03-29T11:31:10.362 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.368+0000 7f44e78c8640 1 -- 172.21.15.114:0/3517462735 >> v1:172.21.15.114:6816/4038755055 conn(0x7f44c406dbe0 legacy=0x7f44c40700a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:10.362 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.368+0000 7f44e78c8640 10 monclient: shutdown 2024-03-29T11:31:10.362 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.368+0000 7f44e78c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:31:10.362 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.368+0000 7f44e78c8640 1 -- 172.21.15.114:0/3517462735 >> v1:172.21.15.114:6790/0 conn(0x7f44e0142690 legacy=0x7f44e01e1c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:10.363 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.368+0000 7f44e78c8640 1 -- 172.21.15.114:0/3517462735 shutdown_connections 2024-03-29T11:31:10.363 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.368+0000 7f44e78c8640 1 -- 172.21.15.114:0/3517462735 >> 172.21.15.114:0/3517462735 conn(0x7f44e0066a00 msgr2=0x7f44e00b2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:10.363 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.369+0000 7f44e78c8640 1 -- 172.21.15.114:0/3517462735 shutdown_connections 2024-03-29T11:31:10.363 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.369+0000 7f44e78c8640 1 -- 172.21.15.114:0/3517462735 wait complete. 2024-03-29T11:31:10.363 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:10.369+0000 7f44e78c8640 1 librados: shutdown 2024-03-29T11:31:10.379 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:31:17.186 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:31:17.358 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.358+0000 7f5312625640 10 monclient: get_monmap_and_config 2024-03-29T11:31:17.358 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.358+0000 7f5312625640 10 monclient: build_initial_monmap 2024-03-29T11:31:17.358 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.358+0000 7f5312625640 10 monclient: monmap: 2024-03-29T11:31:17.358 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:31:17.358 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:31:17.358 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:31:17.359474+0000 2024-03-29T11:31:17.358 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:31:17.359474+0000 2024-03-29T11:31:17.358 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:31:17.358 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:31:17.358 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:31:17.359 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T11:31:17.359 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T11:31:17.359 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:31:17.359 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.359+0000 7f5312625640 1 Processor -- start 2024-03-29T11:31:17.359 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.359+0000 7f5312625640 1 -- start start 2024-03-29T11:31:17.359 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.359+0000 7f5312625640 10 monclient: init 2024-03-29T11:31:17.359 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.360+0000 7f5312625640 10 monclient: _reopen_session rank -1 2024-03-29T11:31:17.359 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.360+0000 7f5312625640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:31:17.359 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.360+0000 7f5312625640 10 monclient(hunting): picked mon.noname-c con 0x7f530c0ad6c0 addr v1:172.21.15.176:6789/0 2024-03-29T11:31:17.359 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.360+0000 7f5312625640 10 monclient(hunting): picked mon.noname-b con 0x7f530c0af6c0 addr v1:172.21.15.114:6790/0 2024-03-29T11:31:17.359 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.360+0000 7f5312625640 10 monclient(hunting): picked mon.noname-a con 0x7f530c14fdc0 addr v1:172.21.15.114:6789/0 2024-03-29T11:31:17.359 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.360+0000 7f5312625640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f530c0a94e0 con 0x7f530c14fdc0 2024-03-29T11:31:17.359 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.360+0000 7f5312625640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f530c0af3a0 con 0x7f530c0ad6c0 2024-03-29T11:31:17.359 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.360+0000 7f5312625640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f530c0afb60 con 0x7f530c0af6c0 2024-03-29T11:31:17.360 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.360+0000 7f5312625640 10 monclient(hunting): _renew_subs 2024-03-29T11:31:17.360 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.360+0000 7f5312625640 10 monclient(hunting): authenticate will time out at 1547.542932s 2024-03-29T11:31:17.360 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.360+0000 7f530bfff640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f530c0ad6c0 0x7f530c152980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.114:38238/0 (socket says 172.21.15.114:38238) 2024-03-29T11:31:17.360 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.360+0000 7f530bfff640 1 -- 172.21.15.114:0/470219658 learned_addr learned my addr 172.21.15.114:0/470219658 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:31:17.360 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.361+0000 7f530affd640 1 -- 172.21.15.114:0/470219658 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2626239745 0 0) 0x7f530c0af3a0 con 0x7f530c0ad6c0 2024-03-29T11:31:17.360 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.361+0000 7f530affd640 10 monclient(hunting): _negotiate 2024-03-29T11:31:17.360 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.361+0000 7f530affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:17.360 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.361+0000 7f530affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:17.360 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.361+0000 7f530affd640 10 monclient(hunting): my global_id is 4561 2024-03-29T11:31:17.360 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.361+0000 7f530affd640 1 -- 172.21.15.114:0/470219658 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52f8003520 con 0x7f530c0ad6c0 2024-03-29T11:31:17.360 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.362+0000 7f530affd640 1 -- 172.21.15.114:0/470219658 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1340159935 0 0) 0x7f52f8003520 con 0x7f530c0ad6c0 2024-03-29T11:31:17.360 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.362+0000 7f530affd640 1 -- 172.21.15.114:0/470219658 >> v1:172.21.15.114:6790/0 conn(0x7f530c0af6c0 legacy=0x7f530c14f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:17.360 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.362+0000 7f530affd640 1 -- 172.21.15.114:0/470219658 >> v1:172.21.15.114:6789/0 conn(0x7f530c14fdc0 legacy=0x7f530c15b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:17.360 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.362+0000 7f530affd640 10 monclient: _finish_hunting 0 2024-03-29T11:31:17.361 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.362+0000 7f530affd640 1 monclient: found mon.noname-c 2024-03-29T11:31:17.361 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.362+0000 7f530affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:31:17.361 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.362+0000 7f530affd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.176:6789/0 2024-03-29T11:31:17.361 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.362+0000 7f530affd640 1 -- 172.21.15.114:0/470219658 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f530c0aab80 con 0x7f530c0ad6c0 2024-03-29T11:31:17.361 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.362+0000 7f530affd640 10 monclient: _finish_auth 0 2024-03-29T11:31:17.361 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.362+0000 7f530affd640 10 monclient: _check_auth_tickets 2024-03-29T11:31:17.361 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.362+0000 7f530affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:31:17.361 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.362+0000 7f530affd640 1 -- 172.21.15.114:0/470219658 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f5304001670 con 0x7f530c0ad6c0 2024-03-29T11:31:17.361 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.362+0000 7f530affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:17.361 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.362+0000 7f530affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:31:17.361 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.362+0000 7f530affd640 10 monclient: dump: 2024-03-29T11:31:17.361 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:31:17.361 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:31:17.361 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:31:17.361 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:31:17.362 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:17.362 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:31:17.362 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:31:17.362 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:31:17.362 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:31:17.362 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:31:17.362 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.362+0000 7f5312625640 5 monclient: authenticate success, global_id 4561 2024-03-29T11:31:17.362 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.362+0000 7f5312625640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:31:17.362 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.362+0000 7f530affd640 1 -- 172.21.15.114:0/470219658 <== mon.1 v1:172.21.15.176:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f53040017d0 con 0x7f530c0ad6c0 2024-03-29T11:31:17.362 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.362+0000 7f530affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:31:17.362 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.362+0000 7f530affd640 1 -- 172.21.15.114:0/470219658 <== mon.1 v1:172.21.15.176:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f5304001c20 con 0x7f530c0ad6c0 2024-03-29T11:31:17.362 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.362+0000 7f530affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:17.362 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.362+0000 7f530affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:31:17.362 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.362+0000 7f530affd640 10 monclient: dump: 2024-03-29T11:31:17.362 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:31:17.363 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:31:17.363 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:31:17.363 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:31:17.363 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:17.363 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:31:17.363 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:31:17.363 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:31:17.363 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:31:17.363 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:31:17.363 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.362+0000 7f5312625640 10 monclient: get_monmap_and_config success 2024-03-29T11:31:17.363 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.362+0000 7f5312625640 4 set_mon_vals no callback set 2024-03-29T11:31:17.363 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.363+0000 7f5312625640 10 monclient: shutdown 2024-03-29T11:31:17.363 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.363+0000 7f5312625640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:31:17.363 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.363+0000 7f5312625640 1 -- 172.21.15.114:0/470219658 >> v1:172.21.15.176:6789/0 conn(0x7f530c0ad6c0 legacy=0x7f530c152980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:17.363 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.363+0000 7f5312625640 1 -- 172.21.15.114:0/470219658 shutdown_connections 2024-03-29T11:31:17.364 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.363+0000 7f5312625640 1 -- 172.21.15.114:0/470219658 >> 172.21.15.114:0/470219658 conn(0x7f530c0b35e0 msgr2=0x7f530c0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:17.364 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.363+0000 7f5312625640 1 -- 172.21.15.114:0/470219658 shutdown_connections 2024-03-29T11:31:17.364 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.364+0000 7f5312625640 1 -- 172.21.15.114:0/470219658 wait complete. 2024-03-29T11:31:17.366 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.364+0000 7f5312625640 10 monclient: build_initial_monmap 2024-03-29T11:31:17.366 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.364+0000 7f5312625640 10 monclient: monmap: 2024-03-29T11:31:17.366 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:31:17.366 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:31:17.366 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T11:31:17.366 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T11:31:17.366 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:31:17.367 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:31:17.367 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:31:17.367 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T11:31:17.367 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T11:31:17.367 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:31:17.367 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.365+0000 7f5312625640 1 librados: starting msgr at 2024-03-29T11:31:17.367 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.365+0000 7f5312625640 1 librados: starting objecter 2024-03-29T11:31:17.367 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.365+0000 7f5312625640 1 Processor -- start 2024-03-29T11:31:17.367 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.365+0000 7f5312625640 1 -- start start 2024-03-29T11:31:17.367 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.365+0000 7f5312625640 1 librados: setting wanted keys 2024-03-29T11:31:17.367 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.365+0000 7f5312625640 1 librados: calling monclient init 2024-03-29T11:31:17.367 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.365+0000 7f5312625640 10 monclient: init 2024-03-29T11:31:17.367 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.366+0000 7f5312625640 10 monclient: _reopen_session rank -1 2024-03-29T11:31:17.367 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.366+0000 7f5312625640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:31:17.367 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.366+0000 7f5312625640 10 monclient(hunting): picked mon.noname-c con 0x7f530c0ad6c0 addr v1:172.21.15.114:6790/0 2024-03-29T11:31:17.368 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.366+0000 7f5312625640 10 monclient(hunting): picked mon.noname-b con 0x7f530c0af6c0 addr v1:172.21.15.176:6789/0 2024-03-29T11:31:17.368 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.367+0000 7f5312625640 10 monclient(hunting): picked mon.noname-a con 0x7f530c14fdc0 addr v1:172.21.15.114:6789/0 2024-03-29T11:31:17.368 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.367+0000 7f5312625640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f530c151d70 con 0x7f530c14fdc0 2024-03-29T11:31:17.368 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.367+0000 7f5312625640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f530c1ed2a0 con 0x7f530c0af6c0 2024-03-29T11:31:17.368 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.367+0000 7f5312625640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f530c152050 con 0x7f530c0ad6c0 2024-03-29T11:31:17.368 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.367+0000 7f5312625640 10 monclient(hunting): _renew_subs 2024-03-29T11:31:17.368 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.367+0000 7f5312625640 10 monclient(hunting): authenticate will time out at 1547.549668s 2024-03-29T11:31:17.368 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.367+0000 7f530bfff640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f530c0ad6c0 0x7f530c150b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:55642/0 (socket says 172.21.15.114:55642) 2024-03-29T11:31:17.368 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.367+0000 7f530bfff640 1 -- 172.21.15.114:0/3514975004 learned_addr learned my addr 172.21.15.114:0/3514975004 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:31:17.368 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.368+0000 7f5308ff9640 10 client.?.objecter ms_handle_connect 0x7f530c0ad6c0 2024-03-29T11:31:17.368 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.368+0000 7f5308ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:31:17.368 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.368+0000 7f5308ff9640 1 -- 172.21.15.114:0/3514975004 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 199105836 0 0) 0x7f530c152050 con 0x7f530c0ad6c0 2024-03-29T11:31:17.368 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.368+0000 7f5308ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:31:17.368 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.368+0000 7f5308ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:17.369 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.368+0000 7f5308ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:17.369 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.368+0000 7f5308ff9640 10 monclient(hunting): my global_id is 4634 2024-03-29T11:31:17.369 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.368+0000 7f5308ff9640 1 -- 172.21.15.114:0/3514975004 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52ec004eb0 con 0x7f530c0ad6c0 2024-03-29T11:31:17.369 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.368+0000 7f5308ff9640 10 client.?.objecter ms_handle_connect 0x7f530c0af6c0 2024-03-29T11:31:17.369 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.369+0000 7f5308ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:31:17.369 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.369+0000 7f5308ff9640 1 -- 172.21.15.114:0/3514975004 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3355363995 0 0) 0x7f530c1ed2a0 con 0x7f530c0af6c0 2024-03-29T11:31:17.369 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.369+0000 7f5308ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:31:17.369 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.369+0000 7f5308ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:17.369 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.369+0000 7f5308ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:17.369 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.369+0000 7f5308ff9640 10 monclient(hunting): my global_id is 4564 2024-03-29T11:31:17.369 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.369+0000 7f5308ff9640 1 -- 172.21.15.114:0/3514975004 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f530c152050 con 0x7f530c0af6c0 2024-03-29T11:31:17.369 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.370+0000 7f5308ff9640 1 -- 172.21.15.114:0/3514975004 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 229041832 0 0) 0x7f530c152050 con 0x7f530c0af6c0 2024-03-29T11:31:17.369 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.370+0000 7f5308ff9640 1 -- 172.21.15.114:0/3514975004 --> v1:172.21.15.176:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f530c1ed2a0 con 0x7f530c0af6c0 2024-03-29T11:31:17.369 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.370+0000 7f5308ff9640 1 -- 172.21.15.114:0/3514975004 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f52f4001930 con 0x7f530c0af6c0 2024-03-29T11:31:17.369 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.370+0000 7f5308ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:17.370 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.370+0000 7f5308ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:31:17.370 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.370+0000 7f5308ff9640 10 monclient(hunting): dump: 2024-03-29T11:31:17.370 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:31:17.370 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:31:17.370 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:31:17.370 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:31:17.370 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:17.370 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:31:17.370 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:31:17.370 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:31:17.370 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:31:17.370 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:31:17.370 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.370+0000 7f5308ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:31:17.370 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.370+0000 7f5308ff9640 10 client.?.objecter ms_handle_connect 0x7f530c14fdc0 2024-03-29T11:31:17.370 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.370+0000 7f5308ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:31:17.371 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.370+0000 7f5308ff9640 1 -- 172.21.15.114:0/3514975004 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 976073938 0 0) 0x7f52ec004eb0 con 0x7f530c0ad6c0 2024-03-29T11:31:17.371 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.370+0000 7f5308ff9640 1 -- 172.21.15.114:0/3514975004 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f530c152050 con 0x7f530c0ad6c0 2024-03-29T11:31:17.371 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.370+0000 7f5308ff9640 1 -- 172.21.15.114:0/3514975004 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f5304001c20 con 0x7f530c0ad6c0 2024-03-29T11:31:17.371 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.370+0000 7f5308ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:17.371 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.370+0000 7f5308ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:31:17.371 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.370+0000 7f5308ff9640 10 monclient(hunting): dump: 2024-03-29T11:31:17.371 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:31:17.371 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:31:17.371 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:31:17.371 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:31:17.371 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:17.371 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:31:17.371 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:31:17.371 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:31:17.372 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:31:17.372 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:31:17.372 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.370+0000 7f5308ff9640 1 -- 172.21.15.114:0/3514975004 <== mon.1 v1:172.21.15.176:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2323748275 0 0) 0x7f530c1ed2a0 con 0x7f530c0af6c0 2024-03-29T11:31:17.372 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.370+0000 7f5308ff9640 1 -- 172.21.15.114:0/3514975004 >> v1:172.21.15.114:6790/0 conn(0x7f530c0ad6c0 legacy=0x7f530c150b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:17.372 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.370+0000 7f5308ff9640 1 -- 172.21.15.114:0/3514975004 >> v1:172.21.15.114:6789/0 conn(0x7f530c14fdc0 legacy=0x7f530c1ecb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:17.372 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.370+0000 7f5308ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:31:17.372 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.370+0000 7f5308ff9640 1 monclient: found mon.b 2024-03-29T11:31:17.372 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.370+0000 7f5308ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:31:17.372 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.370+0000 7f5308ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:31:17.372 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.370+0000 7f5308ff9640 1 -- 172.21.15.114:0/3514975004 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f530c06e9f0 con 0x7f530c0af6c0 2024-03-29T11:31:17.372 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.370+0000 7f5308ff9640 10 monclient: _finish_auth 0 2024-03-29T11:31:17.372 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.370+0000 7f5308ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:31:17.372 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.370+0000 7f5308ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:31:17.372 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.370+0000 7f5312625640 5 monclient: authenticate success, global_id 4564 2024-03-29T11:31:17.372 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.370+0000 7f5312625640 10 monclient: _renew_subs 2024-03-29T11:31:17.373 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.370+0000 7f5312625640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:31:17.373 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.370+0000 7f5312625640 1 -- 172.21.15.114:0/3514975004 --> v1:172.21.15.176:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f530c0b30c0 con 0x7f530c0af6c0 2024-03-29T11:31:17.373 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.370+0000 7f5312625640 10 client.4564.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:31:17.373 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.370+0000 7f5312625640 10 monclient: _renew_subs 2024-03-29T11:31:17.373 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.370+0000 7f5312625640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:31:17.373 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.370+0000 7f5312625640 1 -- 172.21.15.114:0/3514975004 --> v1:172.21.15.176:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f530c151220 con 0x7f530c0af6c0 2024-03-29T11:31:17.373 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.370+0000 7f5312625640 1 librados: init done 2024-03-29T11:31:17.373 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.371+0000 7f5308ff9640 1 -- 172.21.15.114:0/3514975004 <== mon.1 v1:172.21.15.176:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f52f4001a90 con 0x7f530c0af6c0 2024-03-29T11:31:17.373 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.371+0000 7f5308ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:31:17.373 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.371+0000 7f530a7fc640 4 set_mon_vals no callback set 2024-03-29T11:31:17.375 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.372+0000 7f5308ff9640 1 -- 172.21.15.114:0/3514975004 <== mon.1 v1:172.21.15.176:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f52f4003850 con 0x7f530c0af6c0 2024-03-29T11:31:17.375 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.373+0000 7f5308ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:17.375 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.373+0000 7f5308ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:31:17.375 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.373+0000 7f5308ff9640 10 monclient: dump: 2024-03-29T11:31:17.375 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:31:17.375 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:31:17.375 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:31:17.375 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:31:17.375 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:17.375 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:31:17.375 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:31:17.375 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:31:17.375 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:31:17.376 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:31:17.376 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.373+0000 7f5308ff9640 1 -- 172.21.15.114:0/3514975004 <== mon.1 v1:172.21.15.176:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1594292461 0 0) 0x7f52f4003a30 con 0x7f530c0af6c0 2024-03-29T11:31:17.376 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.374+0000 7f5308ff9640 1 -- 172.21.15.114:0/3514975004 <== mon.1 v1:172.21.15.176:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1864569634 0 0) 0x7f52f4083940 con 0x7f530c0af6c0 2024-03-29T11:31:17.376 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.374+0000 7f530bfff640 10 monclient: get_auth_request con 0x7f52ec076650 auth_method 0 2024-03-29T11:31:17.376 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.375+0000 7f5308ff9640 10 client.4564.objecter ms_dispatch 0x7f530c000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:31:17.376 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.375+0000 7f5308ff9640 3 client.4564.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:31:17.376 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.375+0000 7f5308ff9640 3 client.4564.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:31:17.376 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.375+0000 7f5312625640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:17.376 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.375+0000 7f5312625640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:17.376 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.375+0000 7f5312625640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:31:17.376 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.375+0000 7f5308ff9640 20 client.4564.objecter dump_active .. 0 homeless 2024-03-29T11:31:17.376 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.375+0000 7f5312625640 1 -- 172.21.15.114:0/3514975004 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52f0005080 con 0x7f530c0af6c0 2024-03-29T11:31:17.376 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.375+0000 7f5308ff9640 10 client.4564.objecter ms_handle_connect 0x7f52ec076650 2024-03-29T11:31:17.377 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.382+0000 7f5308ff9640 1 -- 172.21.15.114:0/3514975004 <== mon.1 v1:172.21.15.176:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f52f404f050 con 0x7f530c0af6c0 2024-03-29T11:31:17.377 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.382+0000 7f5308ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:17.377 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.382+0000 7f5308ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:31:17.736 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.742+0000 7f5312625640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:31:17.736 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.742+0000 7f5312625640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:31:17.736 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.742+0000 7f5312625640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.176:6789/0 2024-03-29T11:31:17.736 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.742+0000 7f5312625640 1 -- 172.21.15.114:0/3514975004 --> v1:172.21.15.176:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f52f0005330 con 0x7f530c0af6c0 2024-03-29T11:31:17.737 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.743+0000 7f5308ff9640 1 -- 172.21.15.114:0/3514975004 <== mon.1 v1:172.21.15.176:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3515021578) 0x7f52f4051150 con 0x7f530c0af6c0 2024-03-29T11:31:17.737 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.743+0000 7f5308ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:31:17.737 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.743+0000 7f5308ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:31:17.739 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T11:31:17.739 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":11,"fsid":"b0de2b9b-c45a-44c7-b6ee-e83b39a1e108","created":"2024-03-29T11:20:49.420660+0000","modified":"2024-03-29T11:30:54.342208+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:30:54.342208+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a64785d3-6b11-499a-b706-cc827ab96a74","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"5882dadb-e7a8-4604-ba9a-418013c106c8","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"d7e8c409-bc66-4e4a-bbdb-420bad31388e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"a0682989-3595-4cd6-90a5-ab9111c278e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"67068120-e3ff-4817-bd06-44feb7479a35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09e8b-8a85-4997-bdea-75d754e6ee67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2f0d9ead-f47f-4cb3-92a4-0a5ad23cd831","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fdb6a25f-6bd8-4c65-9b9b-b460ba7673a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:31:17.743 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.749+0000 7f5312625640 10 librados: watch_flush enter 2024-03-29T11:31:17.743 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.749+0000 7f5312625640 10 librados: watch_flush exit 2024-03-29T11:31:17.743 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.749+0000 7f5312625640 20 client.4564.objecter shutdown clearing up homeless session... 2024-03-29T11:31:17.743 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.749+0000 7f5312625640 10 client.4564.objecter successfully canceled tick 2024-03-29T11:31:17.743 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.749+0000 7f5312625640 1 -- 172.21.15.114:0/3514975004 >> v1:172.21.15.114:6816/4038755055 conn(0x7f52ec076650 legacy=0x7f52ec078b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:17.743 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.749+0000 7f5312625640 10 monclient: shutdown 2024-03-29T11:31:17.743 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.749+0000 7f5312625640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:31:17.743 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.749+0000 7f5312625640 1 -- 172.21.15.114:0/3514975004 >> v1:172.21.15.176:6789/0 conn(0x7f530c0af6c0 legacy=0x7f530c1ec480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:17.748 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.753+0000 7f5312625640 1 -- 172.21.15.114:0/3514975004 shutdown_connections 2024-03-29T11:31:17.748 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.753+0000 7f5312625640 1 -- 172.21.15.114:0/3514975004 >> 172.21.15.114:0/3514975004 conn(0x7f530c0b35e0 msgr2=0x7f530c0b0900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:17.748 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.754+0000 7f5312625640 1 -- 172.21.15.114:0/3514975004 shutdown_connections 2024-03-29T11:31:17.748 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.754+0000 7f5312625640 1 -- 172.21.15.114:0/3514975004 wait complete. 2024-03-29T11:31:17.748 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:17.754+0000 7f5312625640 1 librados: shutdown 2024-03-29T11:31:17.765 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:31:24.572 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:31:24.724 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.730+0000 7fa49b2f4640 10 monclient: get_monmap_and_config 2024-03-29T11:31:24.724 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.730+0000 7fa49b2f4640 10 monclient: build_initial_monmap 2024-03-29T11:31:24.725 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.730+0000 7fa49b2f4640 10 monclient: monmap: 2024-03-29T11:31:24.725 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:31:24.725 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:31:24.725 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:31:24.731486+0000 2024-03-29T11:31:24.725 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:31:24.731486+0000 2024-03-29T11:31:24.725 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:31:24.725 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:31:24.725 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:31:24.725 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T11:31:24.725 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T11:31:24.725 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:31:24.730 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.731+0000 7fa49b2f4640 1 Processor -- start 2024-03-29T11:31:24.730 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.731+0000 7fa49b2f4640 1 -- start start 2024-03-29T11:31:24.730 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.731+0000 7fa49b2f4640 10 monclient: init 2024-03-29T11:31:24.730 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.732+0000 7fa49b2f4640 10 monclient: _reopen_session rank -1 2024-03-29T11:31:24.730 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.732+0000 7fa49b2f4640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:31:24.730 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.732+0000 7fa49b2f4640 10 monclient(hunting): picked mon.noname-b con 0x7fa4941457f0 addr v1:172.21.15.114:6790/0 2024-03-29T11:31:24.730 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.732+0000 7fa49b2f4640 10 monclient(hunting): picked mon.noname-c con 0x7fa49413f650 addr v1:172.21.15.176:6789/0 2024-03-29T11:31:24.731 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.732+0000 7fa49b2f4640 10 monclient(hunting): picked mon.noname-a con 0x7fa4940a9d70 addr v1:172.21.15.114:6789/0 2024-03-29T11:31:24.731 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.732+0000 7fa49b2f4640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4940aaef0 con 0x7fa4940a9d70 2024-03-29T11:31:24.731 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.732+0000 7fa49b2f4640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa494145bd0 con 0x7fa49413f650 2024-03-29T11:31:24.731 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.732+0000 7fa49b2f4640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa494145eb0 con 0x7fa4941457f0 2024-03-29T11:31:24.731 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.732+0000 7fa49b2f4640 10 monclient(hunting): _renew_subs 2024-03-29T11:31:24.731 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.732+0000 7fa49b2f4640 10 monclient(hunting): authenticate will time out at 1554.914712s 2024-03-29T11:31:24.731 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.732+0000 7fa499891640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fa4940a9d70 0x7fa49414d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:46680/0 (socket says 172.21.15.114:46680) 2024-03-29T11:31:24.731 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.732+0000 7fa499891640 1 -- 172.21.15.114:0/2474431031 learned_addr learned my addr 172.21.15.114:0/2474431031 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:31:24.731 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.733+0000 7fa487fff640 1 -- 172.21.15.114:0/2474431031 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 808647449 0 0) 0x7fa494145eb0 con 0x7fa4941457f0 2024-03-29T11:31:24.731 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.733+0000 7fa487fff640 10 monclient(hunting): _negotiate 2024-03-29T11:31:24.731 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.733+0000 7fa487fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:24.731 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.733+0000 7fa487fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:24.732 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.733+0000 7fa487fff640 10 monclient(hunting): my global_id is 4637 2024-03-29T11:31:24.732 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.733+0000 7fa487fff640 1 -- 172.21.15.114:0/2474431031 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa480003520 con 0x7fa4941457f0 2024-03-29T11:31:24.732 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.733+0000 7fa487fff640 1 -- 172.21.15.114:0/2474431031 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 446128821 0 0) 0x7fa494145bd0 con 0x7fa49413f650 2024-03-29T11:31:24.732 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.733+0000 7fa487fff640 10 monclient(hunting): _negotiate 2024-03-29T11:31:24.732 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.733+0000 7fa487fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:24.732 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.733+0000 7fa487fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:24.732 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.733+0000 7fa487fff640 10 monclient(hunting): my global_id is 4567 2024-03-29T11:31:24.732 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.733+0000 7fa487fff640 1 -- 172.21.15.114:0/2474431031 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa494145eb0 con 0x7fa49413f650 2024-03-29T11:31:24.732 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.733+0000 7fa487fff640 1 -- 172.21.15.114:0/2474431031 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 875753854 0 0) 0x7fa4940aaef0 con 0x7fa4940a9d70 2024-03-29T11:31:24.732 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.733+0000 7fa487fff640 10 monclient(hunting): _negotiate 2024-03-29T11:31:24.732 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.733+0000 7fa487fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:24.732 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.733+0000 7fa487fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:24.732 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.733+0000 7fa487fff640 10 monclient(hunting): my global_id is 4584 2024-03-29T11:31:24.732 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.733+0000 7fa487fff640 1 -- 172.21.15.114:0/2474431031 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa480004a30 con 0x7fa4940a9d70 2024-03-29T11:31:24.732 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.734+0000 7fa487fff640 1 -- 172.21.15.114:0/2474431031 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2468097761 0 0) 0x7fa480004a30 con 0x7fa4940a9d70 2024-03-29T11:31:24.733 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.734+0000 7fa487fff640 1 -- 172.21.15.114:0/2474431031 >> v1:172.21.15.114:6790/0 conn(0x7fa4941457f0 legacy=0x7fa49413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:24.733 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.734+0000 7fa487fff640 1 -- 172.21.15.114:0/2474431031 >> v1:172.21.15.176:6789/0 conn(0x7fa49413f650 legacy=0x7fa4940b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:24.733 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.734+0000 7fa487fff640 10 monclient: _finish_hunting 0 2024-03-29T11:31:24.733 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.734+0000 7fa487fff640 1 monclient: found mon.noname-a 2024-03-29T11:31:24.733 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.734+0000 7fa487fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:31:24.733 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.734+0000 7fa487fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.114:6789/0 2024-03-29T11:31:24.733 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.734+0000 7fa487fff640 1 -- 172.21.15.114:0/2474431031 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa494146340 con 0x7fa4940a9d70 2024-03-29T11:31:24.733 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.734+0000 7fa487fff640 10 monclient: _finish_auth 0 2024-03-29T11:31:24.733 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.734+0000 7fa487fff640 10 monclient: _check_auth_tickets 2024-03-29T11:31:24.733 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.734+0000 7fa487fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:31:24.733 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.734+0000 7fa487fff640 1 -- 172.21.15.114:0/2474431031 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fa48c0019b0 con 0x7fa4940a9d70 2024-03-29T11:31:24.733 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.734+0000 7fa487fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:24.734 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.734+0000 7fa487fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:31:24.734 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.734+0000 7fa487fff640 10 monclient: dump: 2024-03-29T11:31:24.734 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:31:24.734 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:31:24.734 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:31:24.734 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:31:24.734 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:24.734 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:31:24.734 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:31:24.734 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:31:24.734 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:31:24.734 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:31:24.734 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.734+0000 7fa49b2f4640 5 monclient: authenticate success, global_id 4584 2024-03-29T11:31:24.734 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.734+0000 7fa49b2f4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:31:24.735 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.734+0000 7fa487fff640 1 -- 172.21.15.114:0/2474431031 <== mon.0 v1:172.21.15.114:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa48c001b90 con 0x7fa4940a9d70 2024-03-29T11:31:24.735 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.734+0000 7fa487fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:31:24.735 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.734+0000 7fa487fff640 1 -- 172.21.15.114:0/2474431031 <== mon.0 v1:172.21.15.114:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fa48c001f60 con 0x7fa4940a9d70 2024-03-29T11:31:24.735 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.734+0000 7fa487fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:24.735 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.734+0000 7fa487fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:31:24.735 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.734+0000 7fa487fff640 10 monclient: dump: 2024-03-29T11:31:24.735 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:31:24.735 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:31:24.735 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:31:24.735 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:31:24.735 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:24.735 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:31:24.735 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:31:24.736 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:31:24.736 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:31:24.736 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:31:24.736 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.734+0000 7fa49b2f4640 10 monclient: get_monmap_and_config success 2024-03-29T11:31:24.736 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.734+0000 7fa49b2f4640 4 set_mon_vals no callback set 2024-03-29T11:31:24.736 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.735+0000 7fa49b2f4640 10 monclient: shutdown 2024-03-29T11:31:24.736 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.735+0000 7fa49b2f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:31:24.736 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.735+0000 7fa49b2f4640 1 -- 172.21.15.114:0/2474431031 >> v1:172.21.15.114:6789/0 conn(0x7fa4940a9d70 legacy=0x7fa49414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:24.736 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.735+0000 7fa49b2f4640 1 -- 172.21.15.114:0/2474431031 shutdown_connections 2024-03-29T11:31:24.736 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.735+0000 7fa49b2f4640 1 -- 172.21.15.114:0/2474431031 >> 172.21.15.114:0/2474431031 conn(0x7fa494066a00 msgr2=0x7fa494066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:24.736 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.735+0000 7fa49b2f4640 1 -- 172.21.15.114:0/2474431031 shutdown_connections 2024-03-29T11:31:24.736 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.735+0000 7fa49b2f4640 1 -- 172.21.15.114:0/2474431031 wait complete. 2024-03-29T11:31:24.736 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.736+0000 7fa49b2f4640 10 monclient: build_initial_monmap 2024-03-29T11:31:24.737 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.736+0000 7fa49b2f4640 10 monclient: monmap: 2024-03-29T11:31:24.737 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:31:24.737 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:31:24.737 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T11:31:24.737 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T11:31:24.737 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:31:24.737 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:31:24.737 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:31:24.737 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T11:31:24.737 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T11:31:24.737 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:31:24.741 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.736+0000 7fa49b2f4640 1 librados: starting msgr at 2024-03-29T11:31:24.742 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.736+0000 7fa49b2f4640 1 librados: starting objecter 2024-03-29T11:31:24.742 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.737+0000 7fa49b2f4640 1 Processor -- start 2024-03-29T11:31:24.742 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.737+0000 7fa49b2f4640 1 -- start start 2024-03-29T11:31:24.742 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.737+0000 7fa49b2f4640 1 librados: setting wanted keys 2024-03-29T11:31:24.742 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.737+0000 7fa49b2f4640 1 librados: calling monclient init 2024-03-29T11:31:24.742 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.737+0000 7fa49b2f4640 10 monclient: init 2024-03-29T11:31:24.742 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.737+0000 7fa49b2f4640 10 monclient: _reopen_session rank -1 2024-03-29T11:31:24.742 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.737+0000 7fa49b2f4640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:31:24.742 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.737+0000 7fa49b2f4640 10 monclient(hunting): picked mon.noname-b con 0x7fa4940a9d70 addr v1:172.21.15.176:6789/0 2024-03-29T11:31:24.742 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.737+0000 7fa49b2f4640 10 monclient(hunting): picked mon.noname-c con 0x7fa49413f650 addr v1:172.21.15.114:6790/0 2024-03-29T11:31:24.742 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.737+0000 7fa49b2f4640 10 monclient(hunting): picked mon.noname-a con 0x7fa4941457f0 addr v1:172.21.15.114:6789/0 2024-03-29T11:31:24.742 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.737+0000 7fa49b2f4640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4940721f0 con 0x7fa4941457f0 2024-03-29T11:31:24.742 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.737+0000 7fa49b2f4640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa494071f10 con 0x7fa4940a9d70 2024-03-29T11:31:24.742 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.737+0000 7fa49b2f4640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa49413be80 con 0x7fa49413f650 2024-03-29T11:31:24.743 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.737+0000 7fa49b2f4640 10 monclient(hunting): _renew_subs 2024-03-29T11:31:24.743 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.737+0000 7fa49b2f4640 10 monclient(hunting): authenticate will time out at 1554.919971s 2024-03-29T11:31:24.743 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.738+0000 7fa499891640 1 --1- >> v1:172.21.15.114:6789/0 conn(0x7fa4941457f0 0x7fa4940708f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6789/0 says I am v1:172.21.15.114:46984/0 (socket says 172.21.15.114:46984) 2024-03-29T11:31:24.743 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.738+0000 7fa499891640 1 -- 172.21.15.114:0/201999654 learned_addr learned my addr 172.21.15.114:0/201999654 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:31:24.743 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.738+0000 7fa485ffb640 10 client.?.objecter ms_handle_connect 0x7fa4941457f0 2024-03-29T11:31:24.743 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.738+0000 7fa485ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:31:24.743 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.738+0000 7fa485ffb640 10 client.?.objecter ms_handle_connect 0x7fa49413f650 2024-03-29T11:31:24.743 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.738+0000 7fa485ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:31:24.743 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.738+0000 7fa485ffb640 1 -- 172.21.15.114:0/201999654 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2847565975 0 0) 0x7fa4940721f0 con 0x7fa4941457f0 2024-03-29T11:31:24.743 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.739+0000 7fa485ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:31:24.743 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.739+0000 7fa485ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:24.743 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.739+0000 7fa485ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:24.743 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.739+0000 7fa485ffb640 10 monclient(hunting): my global_id is 4587 2024-03-29T11:31:24.743 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.739+0000 7fa485ffb640 1 -- 172.21.15.114:0/201999654 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa478004eb0 con 0x7fa4941457f0 2024-03-29T11:31:24.744 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.739+0000 7fa485ffb640 1 -- 172.21.15.114:0/201999654 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1138831834 0 0) 0x7fa49413be80 con 0x7fa49413f650 2024-03-29T11:31:24.744 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.739+0000 7fa485ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:31:24.744 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.739+0000 7fa485ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:24.744 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.739+0000 7fa485ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:24.744 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.739+0000 7fa485ffb640 10 monclient(hunting): my global_id is 4640 2024-03-29T11:31:24.744 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.739+0000 7fa485ffb640 1 -- 172.21.15.114:0/201999654 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4940721f0 con 0x7fa49413f650 2024-03-29T11:31:24.744 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.739+0000 7fa485ffb640 10 client.?.objecter ms_handle_connect 0x7fa4940a9d70 2024-03-29T11:31:24.744 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.739+0000 7fa485ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:31:24.744 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.739+0000 7fa485ffb640 1 -- 172.21.15.114:0/201999654 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 542062994 0 0) 0x7fa478004eb0 con 0x7fa4941457f0 2024-03-29T11:31:24.744 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.739+0000 7fa485ffb640 1 -- 172.21.15.114:0/201999654 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa49413be80 con 0x7fa4941457f0 2024-03-29T11:31:24.744 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.739+0000 7fa485ffb640 1 -- 172.21.15.114:0/201999654 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fa48c001fa0 con 0x7fa4941457f0 2024-03-29T11:31:24.744 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.739+0000 7fa485ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:24.744 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.739+0000 7fa485ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:31:24.744 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.739+0000 7fa485ffb640 10 monclient(hunting): dump: 2024-03-29T11:31:24.744 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:31:24.745 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:31:24.745 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:31:24.745 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:31:24.745 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:24.745 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:31:24.745 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:31:24.745 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:31:24.745 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:31:24.745 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:31:24.745 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.739+0000 7fa485ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:31:24.745 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.739+0000 7fa485ffb640 1 -- 172.21.15.114:0/201999654 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1757245426 0 0) 0x7fa4940721f0 con 0x7fa49413f650 2024-03-29T11:31:24.745 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.739+0000 7fa485ffb640 1 -- 172.21.15.114:0/201999654 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa478004eb0 con 0x7fa49413f650 2024-03-29T11:31:24.745 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.739+0000 7fa485ffb640 1 -- 172.21.15.114:0/201999654 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fa4880017e0 con 0x7fa49413f650 2024-03-29T11:31:24.745 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.739+0000 7fa485ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:24.745 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.739+0000 7fa485ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:31:24.745 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.739+0000 7fa485ffb640 10 monclient(hunting): dump: 2024-03-29T11:31:24.746 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:31:24.746 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:31:24.746 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:31:24.746 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:31:24.746 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:24.746 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:31:24.746 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:31:24.746 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:31:24.746 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:31:24.746 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:31:24.746 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.739+0000 7fa485ffb640 1 -- 172.21.15.114:0/201999654 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2705000368 0 0) 0x7fa494071f10 con 0x7fa4940a9d70 2024-03-29T11:31:24.746 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.739+0000 7fa485ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:31:24.746 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.739+0000 7fa485ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:24.746 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.739+0000 7fa485ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:24.746 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.739+0000 7fa485ffb640 10 monclient(hunting): my global_id is 4570 2024-03-29T11:31:24.747 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.739+0000 7fa485ffb640 1 -- 172.21.15.114:0/201999654 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4940721f0 con 0x7fa4940a9d70 2024-03-29T11:31:24.747 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.740+0000 7fa485ffb640 1 -- 172.21.15.114:0/201999654 <== mon.0 v1:172.21.15.114:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 838424255 0 0) 0x7fa49413be80 con 0x7fa4941457f0 2024-03-29T11:31:24.747 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.740+0000 7fa485ffb640 1 -- 172.21.15.114:0/201999654 >> v1:172.21.15.114:6790/0 conn(0x7fa49413f650 legacy=0x7fa4940701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:24.747 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.740+0000 7fa485ffb640 1 -- 172.21.15.114:0/201999654 >> v1:172.21.15.176:6789/0 conn(0x7fa4940a9d70 legacy=0x7fa494142580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:24.747 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.740+0000 7fa485ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:31:24.747 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.740+0000 7fa485ffb640 1 monclient: found mon.a 2024-03-29T11:31:24.747 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.740+0000 7fa485ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:31:24.747 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.740+0000 7fa485ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T11:31:24.747 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.740+0000 7fa485ffb640 1 -- 172.21.15.114:0/201999654 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa49413c010 con 0x7fa4941457f0 2024-03-29T11:31:24.747 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.740+0000 7fa485ffb640 10 monclient: _finish_auth 0 2024-03-29T11:31:24.747 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.740+0000 7fa485ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:31:24.747 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.740+0000 7fa485ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:31:24.747 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.740+0000 7fa49b2f4640 5 monclient: authenticate success, global_id 4587 2024-03-29T11:31:24.747 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.740+0000 7fa49b2f4640 10 monclient: _renew_subs 2024-03-29T11:31:24.747 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.740+0000 7fa49b2f4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T11:31:24.748 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.740+0000 7fa49b2f4640 1 -- 172.21.15.114:0/201999654 --> v1:172.21.15.114:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4940b1640 con 0x7fa4941457f0 2024-03-29T11:31:24.748 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.740+0000 7fa49b2f4640 10 client.4587.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:31:24.748 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.740+0000 7fa49b2f4640 10 monclient: _renew_subs 2024-03-29T11:31:24.748 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.740+0000 7fa49b2f4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T11:31:24.748 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.740+0000 7fa49b2f4640 1 -- 172.21.15.114:0/201999654 --> v1:172.21.15.114:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4941e7760 con 0x7fa4941457f0 2024-03-29T11:31:24.748 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.740+0000 7fa49b2f4640 1 librados: init done 2024-03-29T11:31:24.748 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.740+0000 7fa485ffb640 1 -- 172.21.15.114:0/201999654 <== mon.0 v1:172.21.15.114:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa48c004e10 con 0x7fa4941457f0 2024-03-29T11:31:24.748 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.740+0000 7fa485ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:31:24.748 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.740+0000 7fa485ffb640 1 -- 172.21.15.114:0/201999654 <== mon.0 v1:172.21.15.114:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fa48c005120 con 0x7fa4941457f0 2024-03-29T11:31:24.748 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.740+0000 7fa485ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:24.748 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.740+0000 7fa485ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:31:24.748 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.740+0000 7fa485ffb640 10 monclient: dump: 2024-03-29T11:31:24.748 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:31:24.748 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:31:24.748 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:31:24.748 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:31:24.749 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:24.749 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:31:24.749 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:31:24.749 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:31:24.749 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:31:24.749 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:31:24.749 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.740+0000 7fa486ffd640 4 set_mon_vals no callback set 2024-03-29T11:31:24.749 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.742+0000 7fa49b2f4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:24.749 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.742+0000 7fa49b2f4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:24.749 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.742+0000 7fa49b2f4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T11:31:24.749 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.742+0000 7fa49b2f4640 1 -- 172.21.15.114:0/201999654 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa474005080 con 0x7fa4941457f0 2024-03-29T11:31:24.749 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.743+0000 7fa485ffb640 1 -- 172.21.15.114:0/201999654 <== mon.0 v1:172.21.15.114:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1594292461 0 0) 0x7fa48c005280 con 0x7fa4941457f0 2024-03-29T11:31:24.749 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.744+0000 7fa485ffb640 1 -- 172.21.15.114:0/201999654 <== mon.0 v1:172.21.15.114:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1864569634 0 0) 0x7fa48c084fb0 con 0x7fa4941457f0 2024-03-29T11:31:24.749 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.744+0000 7fa485ffb640 10 client.4587.objecter ms_dispatch 0x7fa494000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:31:24.749 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.744+0000 7fa485ffb640 3 client.4587.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:31:24.750 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.744+0000 7fa485ffb640 3 client.4587.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:31:24.750 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.744+0000 7fa485ffb640 20 client.4587.objecter dump_active .. 0 homeless 2024-03-29T11:31:24.750 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.744+0000 7fa499090640 10 monclient: get_auth_request con 0x7fa47806dfe0 auth_method 0 2024-03-29T11:31:24.750 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.745+0000 7fa485ffb640 10 client.4587.objecter ms_handle_connect 0x7fa47806dfe0 2024-03-29T11:31:24.750 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.753+0000 7fa485ffb640 1 -- 172.21.15.114:0/201999654 <== mon.0 v1:172.21.15.114:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa48c050890 con 0x7fa4941457f0 2024-03-29T11:31:24.750 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.753+0000 7fa485ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:24.750 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:24.753+0000 7fa485ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:31:25.109 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:25.115+0000 7fa49b2f4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:31:25.109 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:25.115+0000 7fa49b2f4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:31:25.109 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:25.115+0000 7fa49b2f4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T11:31:25.109 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:25.115+0000 7fa49b2f4640 1 -- 172.21.15.114:0/201999654 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa474005330 con 0x7fa4941457f0 2024-03-29T11:31:25.111 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:25.117+0000 7fa485ffb640 1 -- 172.21.15.114:0/201999654 <== mon.0 v1:172.21.15.114:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3515021578) 0x7fa48c0527c0 con 0x7fa4941457f0 2024-03-29T11:31:25.111 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:25.117+0000 7fa485ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:31:25.111 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:25.117+0000 7fa485ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:31:25.111 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T11:31:25.111 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":11,"fsid":"b0de2b9b-c45a-44c7-b6ee-e83b39a1e108","created":"2024-03-29T11:20:49.420660+0000","modified":"2024-03-29T11:30:54.342208+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:30:54.342208+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a64785d3-6b11-499a-b706-cc827ab96a74","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"5882dadb-e7a8-4604-ba9a-418013c106c8","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"d7e8c409-bc66-4e4a-bbdb-420bad31388e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"a0682989-3595-4cd6-90a5-ab9111c278e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"67068120-e3ff-4817-bd06-44feb7479a35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09e8b-8a85-4997-bdea-75d754e6ee67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2f0d9ead-f47f-4cb3-92a4-0a5ad23cd831","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fdb6a25f-6bd8-4c65-9b9b-b460ba7673a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:31:25.115 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:25.120+0000 7fa49b2f4640 10 librados: watch_flush enter 2024-03-29T11:31:25.115 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:25.121+0000 7fa49b2f4640 10 librados: watch_flush exit 2024-03-29T11:31:25.115 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:25.121+0000 7fa49b2f4640 20 client.4587.objecter shutdown clearing up homeless session... 2024-03-29T11:31:25.115 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:25.121+0000 7fa49b2f4640 10 client.4587.objecter successfully canceled tick 2024-03-29T11:31:25.115 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:25.121+0000 7fa49b2f4640 1 -- 172.21.15.114:0/201999654 >> v1:172.21.15.114:6816/4038755055 conn(0x7fa47806dfe0 legacy=0x7fa4780704a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:25.115 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:25.121+0000 7fa49b2f4640 10 monclient: shutdown 2024-03-29T11:31:25.116 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:25.121+0000 7fa49b2f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:31:25.116 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:25.121+0000 7fa49b2f4640 1 -- 172.21.15.114:0/201999654 >> v1:172.21.15.114:6789/0 conn(0x7fa4941457f0 legacy=0x7fa4940708f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:25.116 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:25.121+0000 7fa49b2f4640 1 -- 172.21.15.114:0/201999654 shutdown_connections 2024-03-29T11:31:25.116 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:25.121+0000 7fa49b2f4640 1 -- 172.21.15.114:0/201999654 >> 172.21.15.114:0/201999654 conn(0x7fa494066a00 msgr2=0x7fa494132a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:25.116 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:25.121+0000 7fa49b2f4640 1 -- 172.21.15.114:0/201999654 shutdown_connections 2024-03-29T11:31:25.116 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:25.121+0000 7fa49b2f4640 1 -- 172.21.15.114:0/201999654 wait complete. 2024-03-29T11:31:25.116 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:25.121+0000 7fa49b2f4640 1 librados: shutdown 2024-03-29T11:31:25.131 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:31:31.938 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:31:32.083 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.088+0000 7f263d234640 10 monclient: get_monmap_and_config 2024-03-29T11:31:32.083 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.088+0000 7f263d234640 10 monclient: build_initial_monmap 2024-03-29T11:31:32.083 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.088+0000 7f263d234640 10 monclient: monmap: 2024-03-29T11:31:32.083 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:31:32.083 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:31:32.083 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:31:32.089737+0000 2024-03-29T11:31:32.083 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:31:32.089737+0000 2024-03-29T11:31:32.083 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:31:32.083 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:31:32.083 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:31:32.083 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T11:31:32.084 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T11:31:32.084 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:31:32.085 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.091+0000 7f263d234640 1 Processor -- start 2024-03-29T11:31:32.088 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.091+0000 7f263d234640 1 -- start start 2024-03-29T11:31:32.089 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.091+0000 7f263d234640 10 monclient: init 2024-03-29T11:31:32.089 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.092+0000 7f263d234640 10 monclient: _reopen_session rank -1 2024-03-29T11:31:32.089 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.092+0000 7f263d234640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:31:32.089 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.092+0000 7f263d234640 10 monclient(hunting): picked mon.noname-a con 0x7f26300389e0 addr v1:172.21.15.114:6789/0 2024-03-29T11:31:32.089 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.092+0000 7f263d234640 10 monclient(hunting): picked mon.noname-b con 0x7f2630038dc0 addr v1:172.21.15.114:6790/0 2024-03-29T11:31:32.089 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.092+0000 7f263d234640 10 monclient(hunting): picked mon.noname-c con 0x7f2630030660 addr v1:172.21.15.176:6789/0 2024-03-29T11:31:32.089 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.092+0000 7f263d234640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26300c54c0 con 0x7f26300389e0 2024-03-29T11:31:32.089 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.092+0000 7f263d234640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2630036430 con 0x7f2630030660 2024-03-29T11:31:32.089 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.092+0000 7f263d234640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2630039840 con 0x7f2630038dc0 2024-03-29T11:31:32.089 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.092+0000 7f263d234640 10 monclient(hunting): _renew_subs 2024-03-29T11:31:32.089 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.092+0000 7f263d234640 10 monclient(hunting): authenticate will time out at 1562.274831s 2024-03-29T11:31:32.090 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.095+0000 7f263ca33640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f2630030660 0x7f26300d3170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.114:48411/0 (socket says 172.21.15.114:48411) 2024-03-29T11:31:32.090 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.095+0000 7f263ca33640 1 -- 172.21.15.114:0/3745511961 learned_addr learned my addr 172.21.15.114:0/3745511961 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:31:32.090 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.096+0000 7f2636ffd640 1 -- 172.21.15.114:0/3745511961 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3083773638 0 0) 0x7f2630036430 con 0x7f2630030660 2024-03-29T11:31:32.091 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.096+0000 7f2636ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:31:32.091 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.096+0000 7f2636ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:32.091 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.096+0000 7f2636ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:32.091 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.096+0000 7f2636ffd640 10 monclient(hunting): my global_id is 4573 2024-03-29T11:31:32.091 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.096+0000 7f2636ffd640 1 -- 172.21.15.114:0/3745511961 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2628003520 con 0x7f2630030660 2024-03-29T11:31:32.094 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.099+0000 7f2636ffd640 1 -- 172.21.15.114:0/3745511961 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 130141717 0 0) 0x7f26300c54c0 con 0x7f26300389e0 2024-03-29T11:31:32.094 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.099+0000 7f2636ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:31:32.094 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.099+0000 7f2636ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:32.094 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.099+0000 7f2636ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:32.094 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.099+0000 7f2636ffd640 10 monclient(hunting): my global_id is 4590 2024-03-29T11:31:32.094 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.100+0000 7f2636ffd640 1 -- 172.21.15.114:0/3745511961 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2630036430 con 0x7f26300389e0 2024-03-29T11:31:32.095 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.100+0000 7f2636ffd640 1 -- 172.21.15.114:0/3745511961 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2484010097 0 0) 0x7f2630036430 con 0x7f26300389e0 2024-03-29T11:31:32.095 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.100+0000 7f2636ffd640 1 -- 172.21.15.114:0/3745511961 >> v1:172.21.15.114:6790/0 conn(0x7f2630038dc0 legacy=0x7f26300cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:32.095 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.100+0000 7f2636ffd640 1 -- 172.21.15.114:0/3745511961 >> v1:172.21.15.176:6789/0 conn(0x7f2630030660 legacy=0x7f26300d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:32.095 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.100+0000 7f2636ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:31:32.095 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.100+0000 7f2636ffd640 1 monclient: found mon.noname-a 2024-03-29T11:31:32.095 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.100+0000 7f2636ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:31:32.095 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.100+0000 7f2636ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.114:6789/0 2024-03-29T11:31:32.095 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.100+0000 7f2636ffd640 1 -- 172.21.15.114:0/3745511961 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2630035690 con 0x7f26300389e0 2024-03-29T11:31:32.096 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.100+0000 7f2636ffd640 10 monclient: _finish_auth 0 2024-03-29T11:31:32.096 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.100+0000 7f2636ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:31:32.096 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.100+0000 7f2636ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:31:32.096 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.100+0000 7f2636ffd640 1 -- 172.21.15.114:0/3745511961 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f2624001690 con 0x7f26300389e0 2024-03-29T11:31:32.096 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.100+0000 7f2636ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:32.096 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.100+0000 7f2636ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:31:32.096 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.100+0000 7f2636ffd640 10 monclient: dump: 2024-03-29T11:31:32.096 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:31:32.096 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:31:32.096 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:31:32.096 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:31:32.096 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:32.096 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:31:32.096 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:31:32.096 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:31:32.096 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:31:32.097 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:31:32.097 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.100+0000 7f2636ffd640 1 -- 172.21.15.114:0/3745511961 <== mon.0 v1:172.21.15.114:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2624001860 con 0x7f26300389e0 2024-03-29T11:31:32.097 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.100+0000 7f2636ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:31:32.097 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.100+0000 7f2636ffd640 1 -- 172.21.15.114:0/3745511961 <== mon.0 v1:172.21.15.114:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f2624001c70 con 0x7f26300389e0 2024-03-29T11:31:32.097 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.100+0000 7f2636ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:32.097 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.100+0000 7f2636ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:31:32.097 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.100+0000 7f2636ffd640 10 monclient: dump: 2024-03-29T11:31:32.097 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:31:32.097 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:31:32.097 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:31:32.097 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:31:32.097 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:32.097 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:31:32.097 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:31:32.097 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:31:32.098 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:31:32.098 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:31:32.098 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.101+0000 7f263d234640 5 monclient: authenticate success, global_id 4590 2024-03-29T11:31:32.098 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.101+0000 7f263d234640 10 monclient: get_monmap_and_config success 2024-03-29T11:31:32.098 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.101+0000 7f263d234640 4 set_mon_vals no callback set 2024-03-29T11:31:32.099 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.102+0000 7f263d234640 10 monclient: shutdown 2024-03-29T11:31:32.099 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.102+0000 7f263d234640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:31:32.099 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.102+0000 7f263d234640 1 -- 172.21.15.114:0/3745511961 >> v1:172.21.15.114:6789/0 conn(0x7f26300389e0 legacy=0x7f2630033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:32.099 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.103+0000 7f263d234640 1 -- 172.21.15.114:0/3745511961 shutdown_connections 2024-03-29T11:31:32.099 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.103+0000 7f263d234640 1 -- 172.21.15.114:0/3745511961 >> 172.21.15.114:0/3745511961 conn(0x7f2630005c00 msgr2=0x7f2630006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:32.099 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.104+0000 7f263d234640 1 -- 172.21.15.114:0/3745511961 shutdown_connections 2024-03-29T11:31:32.099 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.104+0000 7f263d234640 1 -- 172.21.15.114:0/3745511961 wait complete. 2024-03-29T11:31:32.100 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.105+0000 7f263d234640 10 monclient: build_initial_monmap 2024-03-29T11:31:32.100 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.105+0000 7f263d234640 10 monclient: monmap: 2024-03-29T11:31:32.100 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:31:32.100 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:31:32.100 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T11:31:32.100 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T11:31:32.100 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:31:32.100 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:31:32.100 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:31:32.100 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T11:31:32.100 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T11:31:32.100 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:31:32.100 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.105+0000 7f263d234640 1 librados: starting msgr at 2024-03-29T11:31:32.101 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.105+0000 7f263d234640 1 librados: starting objecter 2024-03-29T11:31:32.101 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.105+0000 7f263d234640 1 Processor -- start 2024-03-29T11:31:32.102 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.105+0000 7f263d234640 1 -- start start 2024-03-29T11:31:32.102 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.105+0000 7f263d234640 1 librados: setting wanted keys 2024-03-29T11:31:32.102 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.105+0000 7f263d234640 1 librados: calling monclient init 2024-03-29T11:31:32.102 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.105+0000 7f263d234640 10 monclient: init 2024-03-29T11:31:32.102 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.106+0000 7f263d234640 10 monclient: _reopen_session rank -1 2024-03-29T11:31:32.102 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.106+0000 7f263d234640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:31:32.102 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.106+0000 7f263d234640 10 monclient(hunting): picked mon.noname-a con 0x7f2630030660 addr v1:172.21.15.114:6789/0 2024-03-29T11:31:32.102 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.106+0000 7f263d234640 10 monclient(hunting): picked mon.noname-c con 0x7f26300389e0 addr v1:172.21.15.114:6790/0 2024-03-29T11:31:32.102 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.106+0000 7f263d234640 10 monclient(hunting): picked mon.noname-b con 0x7f26300c23e0 addr v1:172.21.15.176:6789/0 2024-03-29T11:31:32.102 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.106+0000 7f263d234640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f263002d860 con 0x7f2630030660 2024-03-29T11:31:32.102 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.106+0000 7f263d234640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f263000a080 con 0x7f26300c23e0 2024-03-29T11:31:32.103 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.106+0000 7f263d234640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f263002f170 con 0x7f26300389e0 2024-03-29T11:31:32.103 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.106+0000 7f263d234640 10 monclient(hunting): _renew_subs 2024-03-29T11:31:32.103 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.106+0000 7f263d234640 10 monclient(hunting): authenticate will time out at 1562.289030s 2024-03-29T11:31:32.106 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.112+0000 7f26377fe640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f26300389e0 0x7f26300c1cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:35453/0 (socket says 172.21.15.114:35453) 2024-03-29T11:31:32.106 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.112+0000 7f26377fe640 1 -- 172.21.15.114:0/68532972 learned_addr learned my addr 172.21.15.114:0/68532972 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:31:32.107 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.113+0000 7f2634ff9640 10 client.?.objecter ms_handle_connect 0x7f26300c23e0 2024-03-29T11:31:32.107 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.113+0000 7f2634ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:31:32.107 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.113+0000 7f2634ff9640 1 -- 172.21.15.114:0/68532972 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4122056267 0 0) 0x7f263000a080 con 0x7f26300c23e0 2024-03-29T11:31:32.108 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.113+0000 7f2634ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:31:32.108 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.113+0000 7f2634ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:32.108 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.113+0000 7f2634ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:32.108 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.113+0000 7f2634ff9640 10 monclient(hunting): my global_id is 4576 2024-03-29T11:31:32.108 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.113+0000 7f2634ff9640 1 -- 172.21.15.114:0/68532972 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f261c004eb0 con 0x7f26300c23e0 2024-03-29T11:31:32.109 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.115+0000 7f2634ff9640 10 client.?.objecter ms_handle_connect 0x7f2630030660 2024-03-29T11:31:32.109 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.115+0000 7f2634ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:31:32.109 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.115+0000 7f2634ff9640 10 client.?.objecter ms_handle_connect 0x7f26300389e0 2024-03-29T11:31:32.109 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.115+0000 7f2634ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:31:32.110 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.116+0000 7f2634ff9640 1 -- 172.21.15.114:0/68532972 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2200283588 0 0) 0x7f263002f170 con 0x7f26300389e0 2024-03-29T11:31:32.110 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.116+0000 7f2634ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:31:32.110 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.116+0000 7f2634ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:32.110 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.116+0000 7f2634ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:32.110 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.116+0000 7f2634ff9640 10 monclient(hunting): my global_id is 4643 2024-03-29T11:31:32.110 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.116+0000 7f2634ff9640 1 -- 172.21.15.114:0/68532972 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f263000a080 con 0x7f26300389e0 2024-03-29T11:31:32.111 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.116+0000 7f2634ff9640 1 -- 172.21.15.114:0/68532972 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4285772723 0 0) 0x7f263000a080 con 0x7f26300389e0 2024-03-29T11:31:32.111 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.116+0000 7f2634ff9640 1 -- 172.21.15.114:0/68532972 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f263002f170 con 0x7f26300389e0 2024-03-29T11:31:32.111 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.116+0000 7f2634ff9640 1 -- 172.21.15.114:0/68532972 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3116302820 0 0) 0x7f263002d860 con 0x7f2630030660 2024-03-29T11:31:32.111 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.116+0000 7f2634ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:31:32.111 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.116+0000 7f2634ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:32.111 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.116+0000 7f2634ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:32.111 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.116+0000 7f2634ff9640 10 monclient(hunting): my global_id is 4593 2024-03-29T11:31:32.111 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.116+0000 7f2634ff9640 1 -- 172.21.15.114:0/68532972 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f263000a080 con 0x7f2630030660 2024-03-29T11:31:32.111 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.116+0000 7f2634ff9640 1 -- 172.21.15.114:0/68532972 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f262c001990 con 0x7f26300389e0 2024-03-29T11:31:32.112 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.116+0000 7f2634ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:32.112 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.116+0000 7f2634ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:31:32.112 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.116+0000 7f2634ff9640 10 monclient(hunting): dump: 2024-03-29T11:31:32.112 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:31:32.112 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:31:32.112 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:31:32.112 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:31:32.112 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:32.112 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:31:32.112 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:31:32.112 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:31:32.112 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:31:32.112 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:31:32.112 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.116+0000 7f2634ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:31:32.112 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.116+0000 7f2634ff9640 1 -- 172.21.15.114:0/68532972 <== mon.2 v1:172.21.15.114:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 393657507 0 0) 0x7f263002f170 con 0x7f26300389e0 2024-03-29T11:31:32.113 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.116+0000 7f2634ff9640 1 -- 172.21.15.114:0/68532972 >> v1:172.21.15.176:6789/0 conn(0x7f26300c23e0 legacy=0x7f263016c390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:32.113 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.116+0000 7f2634ff9640 1 -- 172.21.15.114:0/68532972 >> v1:172.21.15.114:6789/0 conn(0x7f2630030660 legacy=0x7f263002d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:32.113 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.116+0000 7f2634ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:31:32.113 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.116+0000 7f2634ff9640 1 monclient: found mon.c 2024-03-29T11:31:32.113 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.116+0000 7f2634ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:31:32.113 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.116+0000 7f2634ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:31:32.113 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.116+0000 7f2634ff9640 1 -- 172.21.15.114:0/68532972 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f263002ff30 con 0x7f26300389e0 2024-03-29T11:31:32.113 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.116+0000 7f2634ff9640 10 monclient: _finish_auth 0 2024-03-29T11:31:32.113 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.116+0000 7f2634ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:31:32.113 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.116+0000 7f2634ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:31:32.113 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.117+0000 7f263d234640 5 monclient: authenticate success, global_id 4643 2024-03-29T11:31:32.113 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.117+0000 7f263d234640 10 monclient: _renew_subs 2024-03-29T11:31:32.113 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.117+0000 7f263d234640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:31:32.114 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.117+0000 7f263d234640 1 -- 172.21.15.114:0/68532972 --> v1:172.21.15.114:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26300c74b0 con 0x7f26300389e0 2024-03-29T11:31:32.114 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.117+0000 7f263d234640 10 client.4643.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:31:32.114 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.117+0000 7f263d234640 10 monclient: _renew_subs 2024-03-29T11:31:32.114 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.117+0000 7f263d234640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:31:32.114 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.117+0000 7f263d234640 1 -- 172.21.15.114:0/68532972 --> v1:172.21.15.114:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f263002f690 con 0x7f26300389e0 2024-03-29T11:31:32.114 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.117+0000 7f263d234640 1 librados: init done 2024-03-29T11:31:32.114 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.117+0000 7f2634ff9640 1 -- 172.21.15.114:0/68532972 <== mon.2 v1:172.21.15.114:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f262c001c50 con 0x7f26300389e0 2024-03-29T11:31:32.115 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.117+0000 7f2634ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:31:32.115 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.117+0000 7f2634ff9640 1 -- 172.21.15.114:0/68532972 <== mon.2 v1:172.21.15.114:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f262c003a60 con 0x7f26300389e0 2024-03-29T11:31:32.115 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.117+0000 7f2634ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:32.115 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.117+0000 7f2634ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:31:32.115 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.117+0000 7f2634ff9640 10 monclient: dump: 2024-03-29T11:31:32.115 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:31:32.115 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:31:32.115 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:31:32.115 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:31:32.115 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:32.115 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:31:32.115 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:31:32.115 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:31:32.115 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:31:32.115 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:31:32.116 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.118+0000 7f26367fc640 4 set_mon_vals no callback set 2024-03-29T11:31:32.116 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.120+0000 7f26227fc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:32.116 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.120+0000 7f26227fc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:32.116 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.120+0000 7f26227fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:31:32.116 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.120+0000 7f26227fc640 1 -- 172.21.15.114:0/68532972 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2618005080 con 0x7f26300389e0 2024-03-29T11:31:32.118 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.123+0000 7f2634ff9640 1 -- 172.21.15.114:0/68532972 <== mon.2 v1:172.21.15.114:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1594292461 0 0) 0x7f262c003bc0 con 0x7f26300389e0 2024-03-29T11:31:32.118 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.123+0000 7f2634ff9640 1 -- 172.21.15.114:0/68532972 <== mon.2 v1:172.21.15.114:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1864569634 0 0) 0x7f262c084aa0 con 0x7f26300389e0 2024-03-29T11:31:32.118 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.123+0000 7f2634ff9640 10 client.4643.objecter ms_dispatch 0x7f2638000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:31:32.118 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.123+0000 7f2634ff9640 3 client.4643.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:31:32.118 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.123+0000 7f2634ff9640 3 client.4643.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:31:32.118 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.123+0000 7f2634ff9640 20 client.4643.objecter dump_active .. 0 homeless 2024-03-29T11:31:32.119 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.123+0000 7f2637fff640 10 monclient: get_auth_request con 0x7f261c06db60 auth_method 0 2024-03-29T11:31:32.119 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.124+0000 7f2634ff9640 10 client.4643.objecter ms_handle_connect 0x7f261c06db60 2024-03-29T11:31:32.125 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.131+0000 7f2634ff9640 1 -- 172.21.15.114:0/68532972 <== mon.2 v1:172.21.15.114:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f262c050380 con 0x7f26300389e0 2024-03-29T11:31:32.125 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.131+0000 7f2634ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:32.125 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.131+0000 7f2634ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:31:32.458 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.463+0000 7f26227fc640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:31:32.458 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.463+0000 7f26227fc640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:31:32.458 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.463+0000 7f26227fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:31:32.458 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.463+0000 7f26227fc640 1 -- 172.21.15.114:0/68532972 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2618005330 con 0x7f26300389e0 2024-03-29T11:31:32.458 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.464+0000 7f2634ff9640 1 -- 172.21.15.114:0/68532972 <== mon.2 v1:172.21.15.114:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3515021578) 0x7f262c0522b0 con 0x7f26300389e0 2024-03-29T11:31:32.459 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.464+0000 7f2634ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:31:32.459 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.465+0000 7f2634ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:31:32.459 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T11:31:32.459 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":11,"fsid":"b0de2b9b-c45a-44c7-b6ee-e83b39a1e108","created":"2024-03-29T11:20:49.420660+0000","modified":"2024-03-29T11:30:54.342208+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:30:54.342208+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a64785d3-6b11-499a-b706-cc827ab96a74","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"5882dadb-e7a8-4604-ba9a-418013c106c8","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"d7e8c409-bc66-4e4a-bbdb-420bad31388e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"a0682989-3595-4cd6-90a5-ab9111c278e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"67068120-e3ff-4817-bd06-44feb7479a35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09e8b-8a85-4997-bdea-75d754e6ee67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2f0d9ead-f47f-4cb3-92a4-0a5ad23cd831","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fdb6a25f-6bd8-4c65-9b9b-b460ba7673a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:31:32.463 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.468+0000 7f26227fc640 10 librados: watch_flush enter 2024-03-29T11:31:32.463 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.468+0000 7f26227fc640 10 librados: watch_flush exit 2024-03-29T11:31:32.463 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.468+0000 7f26227fc640 20 client.4643.objecter shutdown clearing up homeless session... 2024-03-29T11:31:32.463 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.468+0000 7f26227fc640 10 client.4643.objecter successfully canceled tick 2024-03-29T11:31:32.463 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.468+0000 7f26227fc640 1 -- 172.21.15.114:0/68532972 >> v1:172.21.15.114:6816/4038755055 conn(0x7f261c06db60 legacy=0x7f261c070020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:32.463 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.468+0000 7f26227fc640 10 monclient: shutdown 2024-03-29T11:31:32.463 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.468+0000 7f26227fc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:31:32.463 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.468+0000 7f26227fc640 1 -- 172.21.15.114:0/68532972 >> v1:172.21.15.114:6790/0 conn(0x7f26300389e0 legacy=0x7f26300c1cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:32.463 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.468+0000 7f26227fc640 1 -- 172.21.15.114:0/68532972 shutdown_connections 2024-03-29T11:31:32.463 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.468+0000 7f26227fc640 1 -- 172.21.15.114:0/68532972 >> 172.21.15.114:0/68532972 conn(0x7f2630005c00 msgr2=0x7f26300ca870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:32.463 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.469+0000 7f26227fc640 1 -- 172.21.15.114:0/68532972 shutdown_connections 2024-03-29T11:31:32.464 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.469+0000 7f26227fc640 1 -- 172.21.15.114:0/68532972 wait complete. 2024-03-29T11:31:32.464 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:32.469+0000 7f26227fc640 1 librados: shutdown 2024-03-29T11:31:32.478 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:31:39.285 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:31:39.434 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.439+0000 7f6a9da84640 10 monclient: get_monmap_and_config 2024-03-29T11:31:39.434 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.439+0000 7f6a9da84640 10 monclient: build_initial_monmap 2024-03-29T11:31:39.434 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.439+0000 7f6a9da84640 10 monclient: monmap: 2024-03-29T11:31:39.434 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:31:39.434 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:31:39.434 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:31:39.440251+0000 2024-03-29T11:31:39.435 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:31:39.440251+0000 2024-03-29T11:31:39.435 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:31:39.435 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:31:39.435 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:31:39.435 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T11:31:39.435 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T11:31:39.435 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:31:39.436 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.442+0000 7f6a9da84640 1 Processor -- start 2024-03-29T11:31:39.436 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.442+0000 7f6a9da84640 1 -- start start 2024-03-29T11:31:39.436 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.442+0000 7f6a9da84640 10 monclient: init 2024-03-29T11:31:39.437 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.443+0000 7f6a9da84640 10 monclient: _reopen_session rank -1 2024-03-29T11:31:39.437 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.443+0000 7f6a9da84640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:31:39.438 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.443+0000 7f6a9da84640 10 monclient(hunting): picked mon.noname-b con 0x7f6a90038d00 addr v1:172.21.15.114:6790/0 2024-03-29T11:31:39.440 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.443+0000 7f6a9da84640 10 monclient(hunting): picked mon.noname-c con 0x7f6a900c49d0 addr v1:172.21.15.176:6789/0 2024-03-29T11:31:39.440 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.444+0000 7f6a9da84640 10 monclient(hunting): picked mon.noname-a con 0x7f6a900c7820 addr v1:172.21.15.114:6789/0 2024-03-29T11:31:39.440 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.444+0000 7f6a9da84640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a9002f790 con 0x7f6a900c7820 2024-03-29T11:31:39.440 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.444+0000 7f6a9da84640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a900307c0 con 0x7f6a900c49d0 2024-03-29T11:31:39.440 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.444+0000 7f6a9da84640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a9002f270 con 0x7f6a90038d00 2024-03-29T11:31:39.440 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.444+0000 7f6a9da84640 10 monclient(hunting): _renew_subs 2024-03-29T11:31:39.440 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.444+0000 7f6a9da84640 10 monclient(hunting): authenticate will time out at 1569.626359s 2024-03-29T11:31:39.440 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.443+0000 7f6a9ca82640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f6a90038d00 0x7f6a9002e960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:38128/0 (socket says 172.21.15.114:38128) 2024-03-29T11:31:39.440 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.444+0000 7f6a9ca82640 1 -- 172.21.15.114:0/1447561051 learned_addr learned my addr 172.21.15.114:0/1447561051 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:31:39.440 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.445+0000 7f6a977fe640 1 -- 172.21.15.114:0/1447561051 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3894969259 0 0) 0x7f6a9002f270 con 0x7f6a90038d00 2024-03-29T11:31:39.440 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.445+0000 7f6a977fe640 10 monclient(hunting): _negotiate 2024-03-29T11:31:39.440 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.445+0000 7f6a977fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:39.440 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.445+0000 7f6a977fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:39.441 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.445+0000 7f6a977fe640 10 monclient(hunting): my global_id is 4646 2024-03-29T11:31:39.441 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.445+0000 7f6a977fe640 1 -- 172.21.15.114:0/1447561051 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a88003520 con 0x7f6a90038d00 2024-03-29T11:31:39.442 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.447+0000 7f6a977fe640 1 -- 172.21.15.114:0/1447561051 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 689504607 0 0) 0x7f6a88003520 con 0x7f6a90038d00 2024-03-29T11:31:39.442 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.447+0000 7f6a977fe640 1 -- 172.21.15.114:0/1447561051 >> v1:172.21.15.176:6789/0 conn(0x7f6a900c49d0 legacy=0x7f6a900c4e30 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T11:31:39.442 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.447+0000 7f6a977fe640 1 -- 172.21.15.114:0/1447561051 >> v1:172.21.15.114:6789/0 conn(0x7f6a900c7820 legacy=0x7f6a900c7c40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:31:39.443 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.448+0000 7f6a977fe640 10 monclient: _finish_hunting 0 2024-03-29T11:31:39.444 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.448+0000 7f6a977fe640 1 monclient: found mon.noname-b 2024-03-29T11:31:39.444 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.448+0000 7f6a977fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:31:39.444 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.449+0000 7f6a977fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.114:6790/0 2024-03-29T11:31:39.444 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.449+0000 7f6a977fe640 1 -- 172.21.15.114:0/1447561051 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a900c8350 con 0x7f6a90038d00 2024-03-29T11:31:39.444 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.449+0000 7f6a977fe640 10 monclient: _finish_auth 0 2024-03-29T11:31:39.444 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.449+0000 7f6a977fe640 10 monclient: _check_auth_tickets 2024-03-29T11:31:39.444 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.449+0000 7f6a977fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:31:39.444 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.449+0000 7f6a977fe640 1 -- 172.21.15.114:0/1447561051 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f6a8c001690 con 0x7f6a90038d00 2024-03-29T11:31:39.444 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.449+0000 7f6a977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:39.444 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.449+0000 7f6a977fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:31:39.444 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.449+0000 7f6a977fe640 10 monclient: dump: 2024-03-29T11:31:39.444 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:31:39.444 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:31:39.444 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:31:39.445 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:31:39.445 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:39.445 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:31:39.445 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:31:39.445 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:31:39.445 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:31:39.445 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:31:39.445 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.451+0000 7f6a9da84640 5 monclient: authenticate success, global_id 4646 2024-03-29T11:31:39.446 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.451+0000 7f6a9da84640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:31:39.446 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.452+0000 7f6a977fe640 1 -- 172.21.15.114:0/1447561051 <== mon.2 v1:172.21.15.114:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6a8c001860 con 0x7f6a90038d00 2024-03-29T11:31:39.447 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.452+0000 7f6a977fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:31:39.447 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.453+0000 7f6a977fe640 1 -- 172.21.15.114:0/1447561051 <== mon.2 v1:172.21.15.114:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f6a8c001cb0 con 0x7f6a90038d00 2024-03-29T11:31:39.447 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.453+0000 7f6a977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:39.451 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.456+0000 7f6a977fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:31:39.451 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.456+0000 7f6a977fe640 10 monclient: dump: 2024-03-29T11:31:39.451 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:31:39.451 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:31:39.451 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:31:39.451 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:31:39.451 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:39.451 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:31:39.452 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:31:39.452 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:31:39.452 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:31:39.452 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:31:39.452 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.456+0000 7f6a9da84640 10 monclient: get_monmap_and_config success 2024-03-29T11:31:39.452 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.456+0000 7f6a9da84640 4 set_mon_vals no callback set 2024-03-29T11:31:39.452 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.457+0000 7f6a9da84640 10 monclient: shutdown 2024-03-29T11:31:39.452 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.457+0000 7f6a9da84640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:31:39.452 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.457+0000 7f6a9da84640 1 -- 172.21.15.114:0/1447561051 >> v1:172.21.15.114:6790/0 conn(0x7f6a90038d00 legacy=0x7f6a9002e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:39.453 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.457+0000 7f6a9da84640 1 -- 172.21.15.114:0/1447561051 shutdown_connections 2024-03-29T11:31:39.453 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.457+0000 7f6a9da84640 1 -- 172.21.15.114:0/1447561051 >> 172.21.15.114:0/1447561051 conn(0x7f6a90005c60 msgr2=0x7f6a90006070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:39.454 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.460+0000 7f6a9da84640 1 -- 172.21.15.114:0/1447561051 shutdown_connections 2024-03-29T11:31:39.456 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.461+0000 7f6a9da84640 1 -- 172.21.15.114:0/1447561051 wait complete. 2024-03-29T11:31:39.458 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.462+0000 7f6a9da84640 10 monclient: build_initial_monmap 2024-03-29T11:31:39.458 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.462+0000 7f6a9da84640 10 monclient: monmap: 2024-03-29T11:31:39.458 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:31:39.459 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:31:39.459 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T11:31:39.459 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T11:31:39.459 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:31:39.459 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:31:39.459 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:31:39.459 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T11:31:39.459 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T11:31:39.459 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:31:39.459 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.463+0000 7f6a9da84640 1 librados: starting msgr at 2024-03-29T11:31:39.459 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.463+0000 7f6a9da84640 1 librados: starting objecter 2024-03-29T11:31:39.459 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.463+0000 7f6a9da84640 1 Processor -- start 2024-03-29T11:31:39.459 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.463+0000 7f6a9da84640 1 -- start start 2024-03-29T11:31:39.459 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.463+0000 7f6a9da84640 1 librados: setting wanted keys 2024-03-29T11:31:39.459 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.463+0000 7f6a9da84640 1 librados: calling monclient init 2024-03-29T11:31:39.460 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.463+0000 7f6a9da84640 10 monclient: init 2024-03-29T11:31:39.460 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.464+0000 7f6a9da84640 10 monclient: _reopen_session rank -1 2024-03-29T11:31:39.460 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.464+0000 7f6a9da84640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:31:39.460 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.464+0000 7f6a9da84640 10 monclient(hunting): picked mon.noname-a con 0x7f6a90038d00 addr v1:172.21.15.114:6789/0 2024-03-29T11:31:39.465 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.464+0000 7f6a9da84640 10 monclient(hunting): picked mon.noname-c con 0x7f6a900c49d0 addr v1:172.21.15.114:6790/0 2024-03-29T11:31:39.465 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.465+0000 7f6a9da84640 10 monclient(hunting): picked mon.noname-b con 0x7f6a900c7820 addr v1:172.21.15.176:6789/0 2024-03-29T11:31:39.465 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.465+0000 7f6a9da84640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a900316f0 con 0x7f6a90038d00 2024-03-29T11:31:39.465 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.465+0000 7f6a9da84640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a9015fe60 con 0x7f6a900c7820 2024-03-29T11:31:39.466 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.465+0000 7f6a9da84640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a900bd350 con 0x7f6a900c49d0 2024-03-29T11:31:39.466 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.465+0000 7f6a9da84640 10 monclient(hunting): _renew_subs 2024-03-29T11:31:39.466 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.465+0000 7f6a9da84640 10 monclient(hunting): authenticate will time out at 1569.647131s 2024-03-29T11:31:39.466 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.465+0000 7f6a97fff640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7f6a900c49d0 0x7f6a900bdba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:53728/0 (socket says 172.21.15.114:53728) 2024-03-29T11:31:39.466 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.465+0000 7f6a97fff640 1 -- 172.21.15.114:0/3397814350 learned_addr learned my addr 172.21.15.114:0/3397814350 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:31:39.466 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.465+0000 7f6a957fa640 10 client.?.objecter ms_handle_connect 0x7f6a900c49d0 2024-03-29T11:31:39.466 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.465+0000 7f6a957fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:31:39.466 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.468+0000 7f6a957fa640 1 -- 172.21.15.114:0/3397814350 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3023574113 0 0) 0x7f6a900bd350 con 0x7f6a900c49d0 2024-03-29T11:31:39.467 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.468+0000 7f6a957fa640 10 monclient(hunting): _negotiate 2024-03-29T11:31:39.467 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.468+0000 7f6a957fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:39.467 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.468+0000 7f6a957fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:39.467 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.468+0000 7f6a957fa640 10 monclient(hunting): my global_id is 4649 2024-03-29T11:31:39.467 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.468+0000 7f6a957fa640 1 -- 172.21.15.114:0/3397814350 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a7c004eb0 con 0x7f6a900c49d0 2024-03-29T11:31:39.467 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.468+0000 7f6a957fa640 10 client.?.objecter ms_handle_connect 0x7f6a900c7820 2024-03-29T11:31:39.467 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.468+0000 7f6a957fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:31:39.467 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.468+0000 7f6a957fa640 10 client.?.objecter ms_handle_connect 0x7f6a90038d00 2024-03-29T11:31:39.467 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.469+0000 7f6a957fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:31:39.467 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.469+0000 7f6a957fa640 1 -- 172.21.15.114:0/3397814350 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 458170461 0 0) 0x7f6a900316f0 con 0x7f6a90038d00 2024-03-29T11:31:39.468 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.469+0000 7f6a957fa640 10 monclient(hunting): _negotiate 2024-03-29T11:31:39.468 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.469+0000 7f6a957fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:39.468 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.469+0000 7f6a957fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:39.468 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.469+0000 7f6a957fa640 10 monclient(hunting): my global_id is 4596 2024-03-29T11:31:39.468 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.469+0000 7f6a957fa640 1 -- 172.21.15.114:0/3397814350 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a900bd350 con 0x7f6a90038d00 2024-03-29T11:31:39.468 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.469+0000 7f6a957fa640 1 -- 172.21.15.114:0/3397814350 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 452970552 0 0) 0x7f6a9015fe60 con 0x7f6a900c7820 2024-03-29T11:31:39.468 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.469+0000 7f6a957fa640 10 monclient(hunting): _negotiate 2024-03-29T11:31:39.468 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.469+0000 7f6a957fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:39.468 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.469+0000 7f6a957fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:39.468 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.469+0000 7f6a957fa640 10 monclient(hunting): my global_id is 4579 2024-03-29T11:31:39.468 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.469+0000 7f6a957fa640 1 -- 172.21.15.114:0/3397814350 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a900316f0 con 0x7f6a900c7820 2024-03-29T11:31:39.474 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.470+0000 7f6a957fa640 1 -- 172.21.15.114:0/3397814350 <== mon.0 v1:172.21.15.114:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3228451326 0 0) 0x7f6a900bd350 con 0x7f6a90038d00 2024-03-29T11:31:39.474 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.470+0000 7f6a957fa640 1 -- 172.21.15.114:0/3397814350 --> v1:172.21.15.114:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6a9015fe60 con 0x7f6a90038d00 2024-03-29T11:31:39.474 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.470+0000 7f6a957fa640 1 -- 172.21.15.114:0/3397814350 <== mon.0 v1:172.21.15.114:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f6a8c001b70 con 0x7f6a90038d00 2024-03-29T11:31:39.474 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.470+0000 7f6a957fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:39.474 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.470+0000 7f6a957fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:31:39.474 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.470+0000 7f6a957fa640 10 monclient(hunting): dump: 2024-03-29T11:31:39.475 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:31:39.475 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:31:39.475 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:31:39.475 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:31:39.475 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:39.475 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:31:39.475 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:31:39.475 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:31:39.475 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:31:39.475 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:31:39.475 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.470+0000 7f6a957fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:31:39.475 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.471+0000 7f6a957fa640 1 -- 172.21.15.114:0/3397814350 <== mon.0 v1:172.21.15.114:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 964075636 0 0) 0x7f6a9015fe60 con 0x7f6a90038d00 2024-03-29T11:31:39.475 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.471+0000 7f6a957fa640 1 -- 172.21.15.114:0/3397814350 >> v1:172.21.15.114:6790/0 conn(0x7f6a900c49d0 legacy=0x7f6a900bdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:39.475 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.471+0000 7f6a957fa640 1 -- 172.21.15.114:0/3397814350 >> v1:172.21.15.176:6789/0 conn(0x7f6a900c7820 legacy=0x7f6a90166e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:39.476 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.471+0000 7f6a957fa640 10 monclient: _finish_hunting 0 2024-03-29T11:31:39.476 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.471+0000 7f6a957fa640 1 monclient: found mon.a 2024-03-29T11:31:39.476 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.471+0000 7f6a957fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:31:39.476 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.471+0000 7f6a957fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T11:31:39.476 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.471+0000 7f6a957fa640 1 -- 172.21.15.114:0/3397814350 --> v1:172.21.15.114:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a9015faf0 con 0x7f6a90038d00 2024-03-29T11:31:39.476 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.471+0000 7f6a957fa640 10 monclient: _finish_auth 0 2024-03-29T11:31:39.476 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.471+0000 7f6a957fa640 10 monclient: _check_auth_tickets 2024-03-29T11:31:39.476 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.471+0000 7f6a957fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:31:39.476 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.471+0000 7f6a957fa640 1 -- 172.21.15.114:0/3397814350 <== mon.0 v1:172.21.15.114:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6a8c004a90 con 0x7f6a90038d00 2024-03-29T11:31:39.476 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.471+0000 7f6a957fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:31:39.476 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.471+0000 7f6a96ffd640 4 set_mon_vals no callback set 2024-03-29T11:31:39.476 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.472+0000 7f6a9da84640 5 monclient: authenticate success, global_id 4596 2024-03-29T11:31:39.476 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.472+0000 7f6a9da84640 10 monclient: _renew_subs 2024-03-29T11:31:39.476 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.472+0000 7f6a9da84640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T11:31:39.476 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.472+0000 7f6a9da84640 1 -- 172.21.15.114:0/3397814350 --> v1:172.21.15.114:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a900be2b0 con 0x7f6a90038d00 2024-03-29T11:31:39.477 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.472+0000 7f6a9da84640 10 client.4596.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:31:39.477 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.472+0000 7f6a9da84640 10 monclient: _renew_subs 2024-03-29T11:31:39.477 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.472+0000 7f6a9da84640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T11:31:39.477 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.472+0000 7f6a9da84640 1 -- 172.21.15.114:0/3397814350 --> v1:172.21.15.114:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a90009a10 con 0x7f6a90038d00 2024-03-29T11:31:39.477 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.472+0000 7f6a9da84640 1 librados: init done 2024-03-29T11:31:39.477 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.472+0000 7f6a957fa640 1 -- 172.21.15.114:0/3397814350 <== mon.0 v1:172.21.15.114:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7f6a8c004da0 con 0x7f6a90038d00 2024-03-29T11:31:39.477 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.472+0000 7f6a957fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:39.477 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.472+0000 7f6a957fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:31:39.477 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.472+0000 7f6a957fa640 10 monclient: dump: 2024-03-29T11:31:39.477 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:31:39.477 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:31:39.477 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:31:39.477 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:31:39.477 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:39.477 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:31:39.478 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:31:39.478 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:31:39.478 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:31:39.478 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:31:39.478 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.473+0000 7f6a9da84640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:39.478 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.473+0000 7f6a9da84640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:39.478 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.473+0000 7f6a9da84640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T11:31:39.478 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.473+0000 7f6a9da84640 1 -- 172.21.15.114:0/3397814350 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a900be5b0 con 0x7f6a90038d00 2024-03-29T11:31:39.478 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.475+0000 7f6a957fa640 1 -- 172.21.15.114:0/3397814350 <== mon.0 v1:172.21.15.114:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1594292461 0 0) 0x7f6a8c01ad70 con 0x7f6a90038d00 2024-03-29T11:31:39.478 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.476+0000 7f6a957fa640 1 -- 172.21.15.114:0/3397814350 <== mon.0 v1:172.21.15.114:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1864569634 0 0) 0x7f6a8c084af0 con 0x7f6a90038d00 2024-03-29T11:31:39.478 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.476+0000 7f6a957fa640 10 client.4596.objecter ms_dispatch 0x7f6a98000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:31:39.478 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.476+0000 7f6a957fa640 3 client.4596.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:31:39.478 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.476+0000 7f6a957fa640 3 client.4596.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:31:39.479 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.476+0000 7f6a957fa640 20 client.4596.objecter dump_active .. 0 homeless 2024-03-29T11:31:39.479 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.477+0000 7f6a97fff640 10 monclient: get_auth_request con 0x7f6a7c076590 auth_method 0 2024-03-29T11:31:39.479 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.477+0000 7f6a957fa640 10 client.4596.objecter ms_handle_connect 0x7f6a7c076590 2024-03-29T11:31:39.479 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.485+0000 7f6a957fa640 1 -- 172.21.15.114:0/3397814350 <== mon.0 v1:172.21.15.114:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6a8c0503d0 con 0x7f6a90038d00 2024-03-29T11:31:39.479 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.485+0000 7f6a957fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:39.479 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.485+0000 7f6a957fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:31:39.802 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.808+0000 7f6a9da84640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:31:39.802 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.808+0000 7f6a9da84640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:31:39.803 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.808+0000 7f6a9da84640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.114:6789/0 2024-03-29T11:31:39.803 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.808+0000 7f6a9da84640 1 -- 172.21.15.114:0/3397814350 --> v1:172.21.15.114:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6a9015fc80 con 0x7f6a90038d00 2024-03-29T11:31:39.803 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.809+0000 7f6a957fa640 1 -- 172.21.15.114:0/3397814350 <== mon.0 v1:172.21.15.114:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3515021578) 0x7f6a9015fc80 con 0x7f6a90038d00 2024-03-29T11:31:39.803 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.809+0000 7f6a957fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:31:39.803 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.809+0000 7f6a957fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:31:39.803 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T11:31:39.804 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":11,"fsid":"b0de2b9b-c45a-44c7-b6ee-e83b39a1e108","created":"2024-03-29T11:20:49.420660+0000","modified":"2024-03-29T11:30:54.342208+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:30:54.342208+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a64785d3-6b11-499a-b706-cc827ab96a74","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"5882dadb-e7a8-4604-ba9a-418013c106c8","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"d7e8c409-bc66-4e4a-bbdb-420bad31388e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"a0682989-3595-4cd6-90a5-ab9111c278e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"67068120-e3ff-4817-bd06-44feb7479a35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09e8b-8a85-4997-bdea-75d754e6ee67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2f0d9ead-f47f-4cb3-92a4-0a5ad23cd831","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fdb6a25f-6bd8-4c65-9b9b-b460ba7673a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:31:39.806 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.812+0000 7f6a9da84640 10 librados: watch_flush enter 2024-03-29T11:31:39.806 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.812+0000 7f6a9da84640 10 librados: watch_flush exit 2024-03-29T11:31:39.806 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.812+0000 7f6a9da84640 20 client.4596.objecter shutdown clearing up homeless session... 2024-03-29T11:31:39.807 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.812+0000 7f6a9da84640 10 client.4596.objecter successfully canceled tick 2024-03-29T11:31:39.807 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.812+0000 7f6a9da84640 1 -- 172.21.15.114:0/3397814350 >> v1:172.21.15.114:6816/4038755055 conn(0x7f6a7c076590 legacy=0x7f6a7c078a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:39.807 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.812+0000 7f6a9da84640 10 monclient: shutdown 2024-03-29T11:31:39.807 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.812+0000 7f6a9da84640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:31:39.807 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.813+0000 7f6a9da84640 1 -- 172.21.15.114:0/3397814350 >> v1:172.21.15.114:6789/0 conn(0x7f6a90038d00 legacy=0x7f6a90160380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:39.808 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.813+0000 7f6a9da84640 1 -- 172.21.15.114:0/3397814350 shutdown_connections 2024-03-29T11:31:39.808 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.814+0000 7f6a9da84640 1 -- 172.21.15.114:0/3397814350 >> 172.21.15.114:0/3397814350 conn(0x7f6a90005c60 msgr2=0x7f6a9002e750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:39.808 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.814+0000 7f6a9da84640 1 -- 172.21.15.114:0/3397814350 shutdown_connections 2024-03-29T11:31:39.808 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.814+0000 7f6a9da84640 1 -- 172.21.15.114:0/3397814350 wait complete. 2024-03-29T11:31:39.808 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:39.814+0000 7f6a9da84640 1 librados: shutdown 2024-03-29T11:31:39.825 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:31:46.632 DEBUG:teuthology.orchestra.run.smithi114:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:31:46.776 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.781+0000 7fe922093640 10 monclient: get_monmap_and_config 2024-03-29T11:31:46.776 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.781+0000 7fe922093640 10 monclient: build_initial_monmap 2024-03-29T11:31:46.777 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.781+0000 7fe922093640 10 monclient: monmap: 2024-03-29T11:31:46.777 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:31:46.777 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:31:46.777 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:31:46.782975+0000 2024-03-29T11:31:46.777 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:31:46.782975+0000 2024-03-29T11:31:46.777 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:31:46.777 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:31:46.777 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:31:46.777 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c 2024-03-29T11:31:46.777 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-b 2024-03-29T11:31:46.777 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:31:46.779 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.783+0000 7fe922093640 1 Processor -- start 2024-03-29T11:31:46.779 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.783+0000 7fe922093640 1 -- start start 2024-03-29T11:31:46.779 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.783+0000 7fe922093640 10 monclient: init 2024-03-29T11:31:46.779 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.784+0000 7fe922093640 10 monclient: _reopen_session rank -1 2024-03-29T11:31:46.779 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.784+0000 7fe922093640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:31:46.779 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.784+0000 7fe922093640 10 monclient(hunting): picked mon.noname-c con 0x7fe91c1422b0 addr v1:172.21.15.176:6789/0 2024-03-29T11:31:46.779 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.784+0000 7fe922093640 10 monclient(hunting): picked mon.noname-b con 0x7fe91c142690 addr v1:172.21.15.114:6790/0 2024-03-29T11:31:46.780 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.784+0000 7fe922093640 10 monclient(hunting): picked mon.noname-a con 0x7fe91c0b17d0 addr v1:172.21.15.114:6789/0 2024-03-29T11:31:46.780 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.784+0000 7fe922093640 1 -- --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe91c142c80 con 0x7fe91c0b17d0 2024-03-29T11:31:46.780 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.784+0000 7fe922093640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe91c0aae40 con 0x7fe91c1422b0 2024-03-29T11:31:46.780 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.784+0000 7fe922093640 1 -- --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe91c13fbd0 con 0x7fe91c142690 2024-03-29T11:31:46.780 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.784+0000 7fe922093640 10 monclient(hunting): _renew_subs 2024-03-29T11:31:46.780 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.784+0000 7fe922093640 10 monclient(hunting): authenticate will time out at 1576.967076s 2024-03-29T11:31:46.781 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.787+0000 7fe91b7fe640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7fe91c1422b0 0x7fe91c13f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.114:53116/0 (socket says 172.21.15.114:53116) 2024-03-29T11:31:46.781 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.787+0000 7fe91b7fe640 1 -- 172.21.15.114:0/1220790167 learned_addr learned my addr 172.21.15.114:0/1220790167 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:31:46.782 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.787+0000 7fe91a7fc640 1 -- 172.21.15.114:0/1220790167 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1559978645 0 0) 0x7fe91c13fbd0 con 0x7fe91c142690 2024-03-29T11:31:46.783 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.787+0000 7fe91a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:31:46.783 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.787+0000 7fe91a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:46.783 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.787+0000 7fe91a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:46.783 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.787+0000 7fe91a7fc640 10 monclient(hunting): my global_id is 4652 2024-03-29T11:31:46.783 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.787+0000 7fe91a7fc640 1 -- 172.21.15.114:0/1220790167 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe904003520 con 0x7fe91c142690 2024-03-29T11:31:46.783 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.788+0000 7fe91a7fc640 1 -- 172.21.15.114:0/1220790167 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3066355262 0 0) 0x7fe91c0aae40 con 0x7fe91c1422b0 2024-03-29T11:31:46.783 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.788+0000 7fe91a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:31:46.783 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.788+0000 7fe91a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:46.783 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.788+0000 7fe91a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:46.783 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.788+0000 7fe91a7fc640 10 monclient(hunting): my global_id is 4582 2024-03-29T11:31:46.784 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.788+0000 7fe91a7fc640 1 -- 172.21.15.114:0/1220790167 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9040048a0 con 0x7fe91c1422b0 2024-03-29T11:31:46.784 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.789+0000 7fe91a7fc640 1 -- 172.21.15.114:0/1220790167 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1393365938 0 0) 0x7fe904003520 con 0x7fe91c142690 2024-03-29T11:31:46.784 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.789+0000 7fe91a7fc640 1 -- 172.21.15.114:0/1220790167 >> v1:172.21.15.176:6789/0 conn(0x7fe91c1422b0 legacy=0x7fe91c13f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:46.784 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.789+0000 7fe91a7fc640 1 -- 172.21.15.114:0/1220790167 >> v1:172.21.15.114:6789/0 conn(0x7fe91c0b17d0 legacy=0x7fe91c0b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:31:46.784 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.790+0000 7fe91a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:31:46.784 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.790+0000 7fe91a7fc640 1 monclient: found mon.noname-b 2024-03-29T11:31:46.784 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.790+0000 7fe91a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:31:46.785 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.790+0000 7fe91a7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.114:6790/0 2024-03-29T11:31:46.785 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.790+0000 7fe91a7fc640 1 -- 172.21.15.114:0/1220790167 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe91c151940 con 0x7fe91c142690 2024-03-29T11:31:46.785 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.790+0000 7fe91a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:31:46.785 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.790+0000 7fe91a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:31:46.785 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.790+0000 7fe91a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:31:46.785 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.790+0000 7fe922093640 5 monclient: authenticate success, global_id 4652 2024-03-29T11:31:46.785 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.790+0000 7fe922093640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:31:46.785 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.790+0000 7fe91a7fc640 1 -- 172.21.15.114:0/1220790167 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fe910001670 con 0x7fe91c142690 2024-03-29T11:31:46.785 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.790+0000 7fe91a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:46.785 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.790+0000 7fe91a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:31:46.785 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.790+0000 7fe91a7fc640 10 monclient: dump: 2024-03-29T11:31:46.785 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:31:46.785 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:31:46.785 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:31:46.786 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:31:46.786 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:46.786 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:31:46.786 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:31:46.786 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:31:46.786 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:31:46.786 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:31:46.786 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.790+0000 7fe922093640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:31:46.787 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.790+0000 7fe91a7fc640 1 -- 172.21.15.114:0/1220790167 <== mon.2 v1:172.21.15.114:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe910001850 con 0x7fe91c142690 2024-03-29T11:31:46.787 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.790+0000 7fe91a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:31:46.787 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.790+0000 7fe922093640 10 monclient: get_monmap_and_config success 2024-03-29T11:31:46.787 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.790+0000 7fe922093640 4 set_mon_vals no callback set 2024-03-29T11:31:46.787 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.790+0000 7fe91a7fc640 1 -- 172.21.15.114:0/1220790167 <== mon.2 v1:172.21.15.114:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fe910001670 con 0x7fe91c142690 2024-03-29T11:31:46.787 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.790+0000 7fe91a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:46.788 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.790+0000 7fe91a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:31:46.788 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.790+0000 7fe91a7fc640 10 monclient: dump: 2024-03-29T11:31:46.788 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:31:46.788 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:31:46.788 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:31:46.788 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:31:46.788 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:46.788 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:31:46.788 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:31:46.788 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:31:46.788 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:31:46.788 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:31:46.788 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.791+0000 7fe922093640 10 monclient: shutdown 2024-03-29T11:31:46.788 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.791+0000 7fe922093640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:31:46.788 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.791+0000 7fe922093640 1 -- 172.21.15.114:0/1220790167 >> v1:172.21.15.114:6790/0 conn(0x7fe91c142690 legacy=0x7fe91c0b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:46.789 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.791+0000 7fe922093640 1 -- 172.21.15.114:0/1220790167 shutdown_connections 2024-03-29T11:31:46.789 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.791+0000 7fe922093640 1 -- 172.21.15.114:0/1220790167 >> 172.21.15.114:0/1220790167 conn(0x7fe91c066a00 msgr2=0x7fe91c066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:46.789 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.792+0000 7fe922093640 1 -- 172.21.15.114:0/1220790167 shutdown_connections 2024-03-29T11:31:46.789 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.792+0000 7fe922093640 1 -- 172.21.15.114:0/1220790167 wait complete. 2024-03-29T11:31:46.790 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.793+0000 7fe922093640 10 monclient: build_initial_monmap 2024-03-29T11:31:46.790 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.793+0000 7fe922093640 10 monclient: monmap: 2024-03-29T11:31:46.790 INFO:teuthology.misc.health.smithi114.stderr:epoch 0 2024-03-29T11:31:46.790 INFO:teuthology.misc.health.smithi114.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:31:46.790 INFO:teuthology.misc.health.smithi114.stderr:last_changed 0.000000 2024-03-29T11:31:46.791 INFO:teuthology.misc.health.smithi114.stderr:created 0.000000 2024-03-29T11:31:46.791 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 0 (unknown) 2024-03-29T11:31:46.791 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:31:46.791 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.noname-a 2024-03-29T11:31:46.791 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.noname-b 2024-03-29T11:31:46.791 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.noname-c 2024-03-29T11:31:46.791 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:31:46.791 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.794+0000 7fe922093640 1 librados: starting msgr at 2024-03-29T11:31:46.791 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.794+0000 7fe922093640 1 librados: starting objecter 2024-03-29T11:31:46.791 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.794+0000 7fe922093640 1 Processor -- start 2024-03-29T11:31:46.791 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.795+0000 7fe922093640 1 -- start start 2024-03-29T11:31:46.791 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.795+0000 7fe922093640 1 librados: setting wanted keys 2024-03-29T11:31:46.791 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.795+0000 7fe922093640 1 librados: calling monclient init 2024-03-29T11:31:46.791 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.795+0000 7fe922093640 10 monclient: init 2024-03-29T11:31:46.792 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.797+0000 7fe922093640 10 monclient: _reopen_session rank -1 2024-03-29T11:31:46.792 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.797+0000 7fe922093640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:31:46.792 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.797+0000 7fe922093640 10 monclient(hunting): picked mon.noname-c con 0x7fe91c0b17d0 addr v1:172.21.15.114:6790/0 2024-03-29T11:31:46.792 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.797+0000 7fe922093640 10 monclient(hunting): picked mon.noname-b con 0x7fe91c1422b0 addr v1:172.21.15.176:6789/0 2024-03-29T11:31:46.795 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.800+0000 7fe91b7fe640 1 --1- >> v1:172.21.15.114:6790/0 conn(0x7fe91c0b17d0 0x7fe91c0ab7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.114:6790/0 says I am v1:172.21.15.114:58773/0 (socket says 172.21.15.114:58773) 2024-03-29T11:31:46.795 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.800+0000 7fe922093640 10 monclient(hunting): picked mon.noname-a con 0x7fe91c142690 addr v1:172.21.15.114:6789/0 2024-03-29T11:31:46.795 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.800+0000 7fe91b7fe640 1 -- 172.21.15.114:0/4155233730 learned_addr learned my addr 172.21.15.114:0/4155233730 (peer_addr_for_me v1:172.21.15.114:0/0) 2024-03-29T11:31:46.795 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.800+0000 7fe922093640 1 -- 172.21.15.114:0/4155233730 --> v1:172.21.15.114:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe91c131e50 con 0x7fe91c142690 2024-03-29T11:31:46.796 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.800+0000 7fe922093640 1 -- 172.21.15.114:0/4155233730 --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe91c132130 con 0x7fe91c1422b0 2024-03-29T11:31:46.796 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.800+0000 7fe922093640 1 -- 172.21.15.114:0/4155233730 --> v1:172.21.15.114:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe91c0ac650 con 0x7fe91c0b17d0 2024-03-29T11:31:46.796 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.800+0000 7fe922093640 10 monclient(hunting): _renew_subs 2024-03-29T11:31:46.796 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.800+0000 7fe922093640 10 monclient(hunting): authenticate will time out at 1576.982747s 2024-03-29T11:31:46.796 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.801+0000 7fe903fff640 10 client.?.objecter ms_handle_connect 0x7fe91c0b17d0 2024-03-29T11:31:46.796 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.801+0000 7fe903fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:31:46.796 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.801+0000 7fe903fff640 10 client.?.objecter ms_handle_connect 0x7fe91c142690 2024-03-29T11:31:46.797 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.801+0000 7fe903fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:31:46.797 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.801+0000 7fe903fff640 1 -- 172.21.15.114:0/4155233730 <== mon.2 v1:172.21.15.114:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1755980135 0 0) 0x7fe91c0ac650 con 0x7fe91c0b17d0 2024-03-29T11:31:46.797 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.801+0000 7fe903fff640 10 monclient(hunting): _negotiate 2024-03-29T11:31:46.797 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.801+0000 7fe903fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:46.797 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.801+0000 7fe903fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:46.797 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.801+0000 7fe903fff640 10 monclient(hunting): my global_id is 4655 2024-03-29T11:31:46.797 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.801+0000 7fe903fff640 1 -- 172.21.15.114:0/4155233730 --> v1:172.21.15.114:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe908004eb0 con 0x7fe91c0b17d0 2024-03-29T11:31:46.797 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.801+0000 7fe903fff640 1 -- 172.21.15.114:0/4155233730 <== mon.0 v1:172.21.15.114:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2022597727 0 0) 0x7fe91c131e50 con 0x7fe91c142690 2024-03-29T11:31:46.797 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.801+0000 7fe903fff640 10 monclient(hunting): _negotiate 2024-03-29T11:31:46.797 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.801+0000 7fe903fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:46.797 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.801+0000 7fe903fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:46.797 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.801+0000 7fe903fff640 10 monclient(hunting): my global_id is 4599 2024-03-29T11:31:46.798 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.801+0000 7fe903fff640 1 -- 172.21.15.114:0/4155233730 --> v1:172.21.15.114:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe91c0ac650 con 0x7fe91c142690 2024-03-29T11:31:46.798 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.802+0000 7fe903fff640 1 -- 172.21.15.114:0/4155233730 <== mon.2 v1:172.21.15.114:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4161245027 0 0) 0x7fe908004eb0 con 0x7fe91c0b17d0 2024-03-29T11:31:46.798 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.802+0000 7fe903fff640 1 -- 172.21.15.114:0/4155233730 --> v1:172.21.15.114:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe91c131e50 con 0x7fe91c0b17d0 2024-03-29T11:31:46.798 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.802+0000 7fe903fff640 1 -- 172.21.15.114:0/4155233730 <== mon.2 v1:172.21.15.114:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fe90c0019f0 con 0x7fe91c0b17d0 2024-03-29T11:31:46.798 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.802+0000 7fe903fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:46.798 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.802+0000 7fe903fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:31:46.798 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.802+0000 7fe903fff640 10 monclient(hunting): dump: 2024-03-29T11:31:46.798 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:31:46.798 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:31:46.798 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:31:46.798 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:31:46.798 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:46.798 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:31:46.798 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:31:46.798 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:31:46.799 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:31:46.799 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:31:46.799 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.802+0000 7fe903fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:31:46.799 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.802+0000 7fe903fff640 10 client.?.objecter ms_handle_connect 0x7fe91c1422b0 2024-03-29T11:31:46.799 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.802+0000 7fe903fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:31:46.799 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.802+0000 7fe903fff640 1 -- 172.21.15.114:0/4155233730 <== mon.2 v1:172.21.15.114:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3334084719 0 0) 0x7fe91c131e50 con 0x7fe91c0b17d0 2024-03-29T11:31:46.799 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.802+0000 7fe903fff640 1 -- 172.21.15.114:0/4155233730 >> v1:172.21.15.176:6789/0 conn(0x7fe91c1422b0 legacy=0x7fe91c12fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:46.799 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.802+0000 7fe903fff640 1 -- 172.21.15.114:0/4155233730 >> v1:172.21.15.114:6789/0 conn(0x7fe91c142690 legacy=0x7fe91c130290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:46.799 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.802+0000 7fe903fff640 10 monclient: _finish_hunting 0 2024-03-29T11:31:46.799 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.802+0000 7fe903fff640 1 monclient: found mon.c 2024-03-29T11:31:46.799 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.802+0000 7fe903fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:31:46.799 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.802+0000 7fe903fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:31:46.799 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.802+0000 7fe903fff640 1 -- 172.21.15.114:0/4155233730 --> v1:172.21.15.114:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe91c0ac930 con 0x7fe91c0b17d0 2024-03-29T11:31:46.799 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.802+0000 7fe903fff640 10 monclient: _finish_auth 0 2024-03-29T11:31:46.799 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.802+0000 7fe903fff640 10 monclient: _check_auth_tickets 2024-03-29T11:31:46.800 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.802+0000 7fe903fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:31:46.800 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.803+0000 7fe922093640 5 monclient: authenticate success, global_id 4655 2024-03-29T11:31:46.800 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.803+0000 7fe922093640 10 monclient: _renew_subs 2024-03-29T11:31:46.801 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.803+0000 7fe922093640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:31:46.801 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.803+0000 7fe922093640 1 -- 172.21.15.114:0/4155233730 --> v1:172.21.15.114:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe91c0acc10 con 0x7fe91c0b17d0 2024-03-29T11:31:46.801 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.803+0000 7fe903fff640 1 -- 172.21.15.114:0/4155233730 <== mon.2 v1:172.21.15.114:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe90c0047e0 con 0x7fe91c0b17d0 2024-03-29T11:31:46.801 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.803+0000 7fe903fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:31:46.801 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.803+0000 7fe903fff640 1 -- 172.21.15.114:0/4155233730 <== mon.2 v1:172.21.15.114:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2009363123 0 0) 0x7fe90c004af0 con 0x7fe91c0b17d0 2024-03-29T11:31:46.801 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.803+0000 7fe903fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:46.801 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.803+0000 7fe903fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:31:46.801 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.803+0000 7fe903fff640 10 monclient: dump: 2024-03-29T11:31:46.801 INFO:teuthology.misc.health.smithi114.stderr:epoch 1 2024-03-29T11:31:46.801 INFO:teuthology.misc.health.smithi114.stderr:fsid b0de2b9b-c45a-44c7-b6ee-e83b39a1e108 2024-03-29T11:31:46.801 INFO:teuthology.misc.health.smithi114.stderr:last_changed 2024-03-29T11:20:11.124295+0000 2024-03-29T11:31:46.801 INFO:teuthology.misc.health.smithi114.stderr:created 2024-03-29T11:20:11.124295+0000 2024-03-29T11:31:46.801 INFO:teuthology.misc.health.smithi114.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:46.801 INFO:teuthology.misc.health.smithi114.stderr:election_strategy: 1 2024-03-29T11:31:46.801 INFO:teuthology.misc.health.smithi114.stderr:0: v1:172.21.15.114:6789/0 mon.a 2024-03-29T11:31:46.802 INFO:teuthology.misc.health.smithi114.stderr:1: v1:172.21.15.176:6789/0 mon.b 2024-03-29T11:31:46.802 INFO:teuthology.misc.health.smithi114.stderr:2: v1:172.21.15.114:6790/0 mon.c 2024-03-29T11:31:46.802 INFO:teuthology.misc.health.smithi114.stderr: 2024-03-29T11:31:46.802 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.804+0000 7fe922093640 10 client.4655.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:31:46.802 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.804+0000 7fe922093640 10 monclient: _renew_subs 2024-03-29T11:31:46.802 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.804+0000 7fe922093640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:31:46.802 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.804+0000 7fe922093640 1 -- 172.21.15.114:0/4155233730 --> v1:172.21.15.114:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe91c1e2e70 con 0x7fe91c0b17d0 2024-03-29T11:31:46.802 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.804+0000 7fe922093640 1 librados: init done 2024-03-29T11:31:46.802 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.804+0000 7fe919ffb640 4 set_mon_vals no callback set 2024-03-29T11:31:46.803 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.805+0000 7fe903fff640 1 -- 172.21.15.114:0/4155233730 <== mon.2 v1:172.21.15.114:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1594292461 0 0) 0x7fe90c01aac0 con 0x7fe91c0b17d0 2024-03-29T11:31:46.803 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.809+0000 7fe903fff640 1 -- 172.21.15.114:0/4155233730 <== mon.2 v1:172.21.15.114:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1864569634 0 0) 0x7fe90c054990 con 0x7fe91c0b17d0 2024-03-29T11:31:46.803 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.809+0000 7fe903fff640 10 client.4655.objecter ms_dispatch 0x7fe91c000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:31:46.803 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.809+0000 7fe903fff640 3 client.4655.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:31:46.803 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.809+0000 7fe903fff640 3 client.4655.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:31:46.803 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.809+0000 7fe903fff640 20 client.4655.objecter dump_active .. 0 homeless 2024-03-29T11:31:46.803 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.809+0000 7fe91affd640 10 monclient: get_auth_request con 0x7fe90806dd10 auth_method 0 2024-03-29T11:31:46.804 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.809+0000 7fe903fff640 10 client.4655.objecter ms_handle_connect 0x7fe90806dd10 2024-03-29T11:31:46.804 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.809+0000 7fe922093640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:46.804 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.809+0000 7fe922093640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:46.804 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.809+0000 7fe922093640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:31:46.804 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.809+0000 7fe922093640 1 -- 172.21.15.114:0/4155233730 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe91c075f10 con 0x7fe91c0b17d0 2024-03-29T11:31:46.813 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.819+0000 7fe903fff640 1 -- 172.21.15.114:0/4155233730 <== mon.2 v1:172.21.15.114:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe90c0502d0 con 0x7fe91c0b17d0 2024-03-29T11:31:46.813 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.819+0000 7fe903fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:46.814 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:46.819+0000 7fe903fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:31:47.129 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:47.135+0000 7fe922093640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:31:47.130 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:47.135+0000 7fe922093640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:31:47.130 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:47.135+0000 7fe922093640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.114:6790/0 2024-03-29T11:31:47.130 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:47.135+0000 7fe922093640 1 -- 172.21.15.114:0/4155233730 --> v1:172.21.15.114:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe91c0b2a30 con 0x7fe91c0b17d0 2024-03-29T11:31:47.130 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:47.136+0000 7fe903fff640 1 -- 172.21.15.114:0/4155233730 <== mon.2 v1:172.21.15.114:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3515021578) 0x7fe90c0523d0 con 0x7fe91c0b17d0 2024-03-29T11:31:47.130 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:47.136+0000 7fe903fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:31:47.130 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:47.136+0000 7fe903fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:31:47.132 INFO:teuthology.misc.health.smithi114.stdout: 2024-03-29T11:31:47.132 INFO:teuthology.misc.health.smithi114.stdout:{"epoch":11,"fsid":"b0de2b9b-c45a-44c7-b6ee-e83b39a1e108","created":"2024-03-29T11:20:49.420660+0000","modified":"2024-03-29T11:30:54.342208+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:30:54.342208+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a64785d3-6b11-499a-b706-cc827ab96a74","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"5882dadb-e7a8-4604-ba9a-418013c106c8","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"d7e8c409-bc66-4e4a-bbdb-420bad31388e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"a0682989-3595-4cd6-90a5-ab9111c278e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"67068120-e3ff-4817-bd06-44feb7479a35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ecd09e8b-8a85-4997-bdea-75d754e6ee67","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2f0d9ead-f47f-4cb3-92a4-0a5ad23cd831","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fdb6a25f-6bd8-4c65-9b9b-b460ba7673a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:31:47.134 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:47.139+0000 7fe922093640 10 librados: watch_flush enter 2024-03-29T11:31:47.134 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:47.140+0000 7fe922093640 10 librados: watch_flush exit 2024-03-29T11:31:47.134 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:47.140+0000 7fe922093640 20 client.4655.objecter shutdown clearing up homeless session... 2024-03-29T11:31:47.134 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:47.140+0000 7fe922093640 10 client.4655.objecter successfully canceled tick 2024-03-29T11:31:47.135 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:47.140+0000 7fe922093640 1 -- 172.21.15.114:0/4155233730 >> v1:172.21.15.114:6816/4038755055 conn(0x7fe90806dd10 legacy=0x7fe9080701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:47.135 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:47.140+0000 7fe922093640 10 monclient: shutdown 2024-03-29T11:31:47.135 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:47.140+0000 7fe922093640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:31:47.135 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:47.140+0000 7fe922093640 1 -- 172.21.15.114:0/4155233730 >> v1:172.21.15.114:6790/0 conn(0x7fe91c0b17d0 legacy=0x7fe91c0ab7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:47.135 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:47.140+0000 7fe922093640 1 -- 172.21.15.114:0/4155233730 shutdown_connections 2024-03-29T11:31:47.135 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:47.140+0000 7fe922093640 1 -- 172.21.15.114:0/4155233730 >> 172.21.15.114:0/4155233730 conn(0x7fe91c066a00 msgr2=0x7fe91c13c4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:47.135 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:47.140+0000 7fe922093640 1 -- 172.21.15.114:0/4155233730 shutdown_connections 2024-03-29T11:31:47.135 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:47.140+0000 7fe922093640 1 -- 172.21.15.114:0/4155233730 wait complete. 2024-03-29T11:31:47.135 INFO:teuthology.misc.health.smithi114.stderr:2024-03-29T11:31:47.140+0000 7fe922093640 1 librados: shutdown 2024-03-29T11:31:47.154 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:31:47.154 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T11:31:47.156 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-29T11:31:47.156 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-29T11:31:47.156 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:31:47.194 INFO:tasks.ceph.osd.0:Stopped 2024-03-29T11:31:47.194 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-29T11:31:47.194 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:31:47.271 INFO:tasks.ceph.osd.1:Stopped 2024-03-29T11:31:47.272 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-29T11:31:47.272 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:31:47.347 INFO:tasks.ceph.osd.2:Stopped 2024-03-29T11:31:47.347 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-29T11:31:47.348 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:31:47.427 INFO:tasks.ceph.osd.3:Stopped 2024-03-29T11:31:47.427 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-29T11:31:47.428 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:31:47.467 INFO:tasks.ceph.osd.4:Stopped 2024-03-29T11:31:47.467 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-29T11:31:47.467 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:31:47.542 INFO:tasks.ceph.osd.5:Stopped 2024-03-29T11:31:47.542 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-29T11:31:47.543 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:31:47.616 INFO:tasks.ceph.osd.6:Stopped 2024-03-29T11:31:47.616 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-29T11:31:47.616 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:31:47.687 INFO:tasks.ceph.osd.7:Stopped 2024-03-29T11:31:47.687 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-29T11:31:47.688 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-29T11:31:47.688 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:31:47.713 INFO:tasks.ceph.mgr.y:Stopped 2024-03-29T11:31:47.713 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-29T11:31:47.713 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:31:47.753 INFO:tasks.ceph.mgr.x:Stopped 2024-03-29T11:31:47.753 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-29T11:31:47.753 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-29T11:31:47.753 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:31:47.766 INFO:tasks.ceph.mon.a:Stopped 2024-03-29T11:31:47.767 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-29T11:31:47.767 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:31:47.818 INFO:tasks.ceph.mon.c:Stopped 2024-03-29T11:31:47.818 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-29T11:31:47.818 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:31:47.829 INFO:tasks.ceph.mon.b:Stopped 2024-03-29T11:31:47.830 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-29T11:31:47.830 DEBUG:teuthology.orchestra.run.smithi114:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | head -n 1 2024-03-29T11:31:47.891 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi114.front.sepia.ceph.com 2024-03-29T11:31:47.891 DEBUG:teuthology.orchestra.run.smithi114:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-29T11:31:50.614 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi114.front.sepia.ceph.com 2024-03-29T11:31:50.614 DEBUG:teuthology.orchestra.run.smithi114:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-29T11:31:50.733 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi114.front.sepia.ceph.com 2024-03-29T11:31:50.734 DEBUG:teuthology.orchestra.run.smithi114:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-29T11:31:50.793 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi114.front.sepia.ceph.com 2024-03-29T11:31:50.794 DEBUG:teuthology.orchestra.run.smithi114:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-29T11:31:50.850 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi176.front.sepia.ceph.com 2024-03-29T11:31:50.851 DEBUG:teuthology.orchestra.run.smithi176:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-29T11:31:54.349 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi176.front.sepia.ceph.com 2024-03-29T11:31:54.349 DEBUG:teuthology.orchestra.run.smithi176:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-29T11:31:54.419 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi176.front.sepia.ceph.com 2024-03-29T11:31:54.419 DEBUG:teuthology.orchestra.run.smithi176:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-29T11:31:54.483 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi176.front.sepia.ceph.com 2024-03-29T11:31:54.484 DEBUG:teuthology.orchestra.run.smithi176:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-29T11:31:54.535 INFO:tasks.ceph:Archiving mon data... 2024-03-29T11:31:54.537 DEBUG:teuthology.misc:Transferring archived files from smithi114:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622690/data/mon.a.tgz 2024-03-29T11:31:54.537 DEBUG:teuthology.orchestra.run.smithi114:> mktemp 2024-03-29T11:31:54.556 INFO:teuthology.orchestra.run.smithi114.stdout:/tmp/tmp.bnrF7FJC8Y 2024-03-29T11:31:54.556 DEBUG:teuthology.orchestra.run.smithi114:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.bnrF7FJC8Y 2024-03-29T11:31:54.663 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 0666 /tmp/tmp.bnrF7FJC8Y 2024-03-29T11:31:54.770 DEBUG:teuthology.orchestra.remote:smithi114:/tmp/tmp.bnrF7FJC8Y is 127KB 2024-03-29T11:31:54.806 DEBUG:teuthology.orchestra.run.smithi114:> rm -fr /tmp/tmp.bnrF7FJC8Y 2024-03-29T11:31:54.822 DEBUG:teuthology.misc:Transferring archived files from smithi114:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622690/data/mon.c.tgz 2024-03-29T11:31:54.823 DEBUG:teuthology.orchestra.run.smithi114:> mktemp 2024-03-29T11:31:54.881 INFO:teuthology.orchestra.run.smithi114.stdout:/tmp/tmp.NsU8nZBojx 2024-03-29T11:31:54.882 DEBUG:teuthology.orchestra.run.smithi114:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.NsU8nZBojx 2024-03-29T11:31:54.972 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 0666 /tmp/tmp.NsU8nZBojx 2024-03-29T11:31:55.062 DEBUG:teuthology.orchestra.remote:smithi114:/tmp/tmp.NsU8nZBojx is 127KB 2024-03-29T11:31:55.087 DEBUG:teuthology.orchestra.run.smithi114:> rm -fr /tmp/tmp.NsU8nZBojx 2024-03-29T11:31:55.101 DEBUG:teuthology.misc:Transferring archived files from smithi176:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622690/data/mon.b.tgz 2024-03-29T11:31:55.102 DEBUG:teuthology.orchestra.run.smithi176:> mktemp 2024-03-29T11:31:55.120 INFO:teuthology.orchestra.run.smithi176.stdout:/tmp/tmp.fYl8aaPR3z 2024-03-29T11:31:55.120 DEBUG:teuthology.orchestra.run.smithi176:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.fYl8aaPR3z 2024-03-29T11:31:55.228 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod 0666 /tmp/tmp.fYl8aaPR3z 2024-03-29T11:31:55.317 DEBUG:teuthology.orchestra.remote:smithi176:/tmp/tmp.fYl8aaPR3z is 127KB 2024-03-29T11:31:55.368 DEBUG:teuthology.orchestra.run.smithi176:> rm -fr /tmp/tmp.fYl8aaPR3z 2024-03-29T11:31:55.383 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-29T11:31:55.383 DEBUG:teuthology.orchestra.run.smithi114:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-29T11:31:55.386 DEBUG:teuthology.orchestra.run.smithi176:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-29T11:31:55.467 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-29T11:31:55.508 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-29T11:31:55.508 INFO:tasks.ceph:Archiving crash dumps... 2024-03-29T11:31:55.510 DEBUG:teuthology.misc:Transferring archived files from smithi114:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622690/remote/smithi114/crash 2024-03-29T11:31:55.510 DEBUG:teuthology.orchestra.run.smithi114:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-29T11:31:55.537 DEBUG:teuthology.misc:Transferring archived files from smithi176:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622690/remote/smithi176/crash 2024-03-29T11:31:55.538 DEBUG:teuthology.orchestra.run.smithi176:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-29T11:31:55.576 INFO:tasks.ceph:Compressing logs... 2024-03-29T11:31:55.576 DEBUG:teuthology.orchestra.run.smithi114:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-29T11:31:55.581 DEBUG:teuthology.orchestra.run.smithi176:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-29T11:31:55.641 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40158.log 2024-03-29T11:31:55.641 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40392.log 2024-03-29T11:31:55.642 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.40158.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41723.log 2024-03-29T11:31:55.642 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.40392.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39963.log 2024-03-29T11:31:55.642 INFO:teuthology.orchestra.run.smithi114.stderr: 85.3% -- replaced with /var/log/ceph/ceph-client.admin.40158.log.gz 2024-03-29T11:31:55.642 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.41723.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-29T11:31:55.642 INFO:teuthology.orchestra.run.smithi114.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.40392.log.gz 2024-03-29T11:31:55.642 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41173.log 2024-03-29T11:31:55.642 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.39963.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41723.log.gz 2024-03-29T11:31:55.642 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38396.log 2024-03-29T11:31:55.642 INFO:teuthology.orchestra.run.smithi114.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.39963.log.gz 2024-03-29T11:31:55.642 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.41173.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41762.log 2024-03-29T11:31:55.642 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.38396.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40197.log 2024-03-29T11:31:55.642 INFO:teuthology.orchestra.run.smithi114.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41173.log.gz 2024-03-29T11:31:55.642 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.41762.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38435.log 2024-03-29T11:31:55.643 INFO:teuthology.orchestra.run.smithi114.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.38396.log.gz 2024-03-29T11:31:55.643 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.40197.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.41762.log.gz 2024-03-29T11:31:55.643 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41410.log 2024-03-29T11:31:55.643 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.38435.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39841.log 2024-03-29T11:31:55.643 INFO:teuthology.orchestra.run.smithi114.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.40197.log.gz 2024-03-29T11:31:55.643 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.41410.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38632.log 2024-03-29T11:31:55.643 INFO:teuthology.orchestra.run.smithi114.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.38435.log.gz 2024-03-29T11:31:55.643 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.39841.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40900.log 2024-03-29T11:31:55.643 INFO:teuthology.orchestra.run.smithi114.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.41410.log.gz 2024-03-29T11:31:55.643 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.38632.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34957.log 2024-03-29T11:31:55.643 INFO:teuthology.orchestra.run.smithi114.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.39841.log.gz 2024-03-29T11:31:55.643 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.40900.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41329.log 2024-03-29T11:31:55.643 INFO:teuthology.orchestra.run.smithi114.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.38632.log.gz 2024-03-29T11:31:55.643 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.34957.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40900.log.gz 2024-03-29T11:31:55.643 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41684.log 2024-03-29T11:31:55.644 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40002.log 2024-03-29T11:31:55.644 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.41329.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.34957.log.gz 2024-03-29T11:31:55.644 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.41684.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41134.log 2024-03-29T11:31:55.644 INFO:teuthology.orchestra.run.smithi114.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.41329.log.gz 2024-03-29T11:31:55.644 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.40002.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41567.log 2024-03-29T11:31:55.644 INFO:teuthology.orchestra.run.smithi114.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.41684.log.gz 2024-03-29T11:31:55.644 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.41134.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39607.log 2024-03-29T11:31:55.644 INFO:teuthology.orchestra.run.smithi114.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40002.log.gz 2024-03-29T11:31:55.644 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.41567.log: 86.1%gzip -- replaced with /var/log/ceph/ceph-client.admin.41134.log.gz 2024-03-29T11:31:55.644 INFO:teuthology.orchestra.run.smithi114.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.40588.log 2024-03-29T11:31:55.644 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.39607.log: gzip 85.9% -5 -- replaced with /var/log/ceph/ceph-client.admin.41567.log.gz --verbose 2024-03-29T11:31:55.644 INFO:teuthology.orchestra.run.smithi114.stderr: -- /var/log/ceph/ceph-client.admin.39646.log 2024-03-29T11:31:55.644 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.40588.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39685.log 2024-03-29T11:31:55.644 INFO:teuthology.orchestra.run.smithi114.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.39607.log.gz 2024-03-29T11:31:55.644 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.39646.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38591.log 2024-03-29T11:31:55.645 INFO:teuthology.orchestra.run.smithi114.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40588.log.gz 2024-03-29T11:31:55.645 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.39685.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39568.log 2024-03-29T11:31:55.645 INFO:teuthology.orchestra.run.smithi114.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39646.log.gz 2024-03-29T11:31:55.645 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.38591.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39763.log 2024-03-29T11:31:55.645 INFO:teuthology.orchestra.run.smithi114.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39685.log.gz 2024-03-29T11:31:55.645 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.39568.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.38591.log.gz 2024-03-29T11:31:55.645 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38710.log 2024-03-29T11:31:55.645 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.39763.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39373.log 2024-03-29T11:31:55.645 INFO:teuthology.orchestra.run.smithi114.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.39568.log.gz 2024-03-29T11:31:55.645 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.38710.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40861.log 2024-03-29T11:31:55.645 INFO:teuthology.orchestra.run.smithi114.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.39763.log.gz 2024-03-29T11:31:55.645 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.39373.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.38710.log.gz 2024-03-29T11:31:55.646 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39217.log 2024-03-29T11:31:55.646 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38788.log 2024-03-29T11:31:55.646 INFO:teuthology.orchestra.run.smithi114.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39373.log.gz 2024-03-29T11:31:55.646 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.40861.log: /var/log/ceph/ceph-client.admin.39217.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41017.log 2024-03-29T11:31:55.646 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.38788.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40549.log 2024-03-29T11:31:55.646 INFO:teuthology.orchestra.run.smithi114.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.39217.log.gz 2024-03-29T11:31:55.646 INFO:teuthology.orchestra.run.smithi114.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40861.log.gz 2024-03-29T11:31:55.646 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.41017.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-29T11:31:55.646 INFO:teuthology.orchestra.run.smithi114.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.38788.log.gz 2024-03-29T11:31:55.646 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.40549.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39724.log 2024-03-29T11:31:55.646 INFO:teuthology.orchestra.run.smithi114.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41017.log.gz 2024-03-29T11:31:55.646 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph.audit.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40549.log.gz 2024-03-29T11:31:55.647 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38357.log 2024-03-29T11:31:55.647 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.39724.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38474.log 2024-03-29T11:31:55.647 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.38357.log: 93.5% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-29T11:31:55.647 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39295.log 2024-03-29T11:31:55.647 INFO:teuthology.orchestra.run.smithi114.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39724.log.gz 2024-03-29T11:31:55.647 INFO:teuthology.orchestra.run.smithi114.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.38357.log.gz 2024-03-29T11:31:55.647 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.38474.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38983.log 2024-03-29T11:31:55.647 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.39295.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.28859.log 2024-03-29T11:31:55.647 INFO:teuthology.orchestra.run.smithi114.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.38474.log.gz 2024-03-29T11:31:55.647 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.38983.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41056.log 2024-03-29T11:31:55.647 INFO:teuthology.orchestra.run.smithi114.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39295.log.gz 2024-03-29T11:31:55.647 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph.tmp-client.admin.28859.log: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.38983.log.gz 2024-03-29T11:31:55.647 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40939.log 2024-03-29T11:31:55.647 INFO:teuthology.orchestra.run.smithi114.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.28859.log.gz 2024-03-29T11:31:55.647 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.41056.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41368.log 2024-03-29T11:31:55.648 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.40939.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41212.log 2024-03-29T11:31:55.648 INFO:teuthology.orchestra.run.smithi114.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.41056.log.gz 2024-03-29T11:31:55.648 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.41368.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.40939.log.gz 2024-03-29T11:31:55.648 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-29T11:31:55.648 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.41212.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40119.log 2024-03-29T11:31:55.648 INFO:teuthology.orchestra.run.smithi114.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.41368.log.gz 2024-03-29T11:31:55.648 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40314.log 2024-03-29T11:31:55.648 INFO:teuthology.orchestra.run.smithi114.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41212.log.gz 2024-03-29T11:31:55.648 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.40119.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39256.log 2024-03-29T11:31:55.648 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.40314.log: gzip -5 85.8% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.40119.log.gz 2024-03-29T11:31:55.648 INFO:teuthology.orchestra.run.smithi114.stderr: /var/log/ceph/ceph-client.admin.38552.log 2024-03-29T11:31:55.648 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38944.log 2024-03-29T11:31:55.648 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.39256.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.40314.log.gz 2024-03-29T11:31:55.648 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.38552.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39061.log 2024-03-29T11:31:55.648 INFO:teuthology.orchestra.run.smithi114.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.39256.log.gz 2024-03-29T11:31:55.649 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.38944.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.38552.log.gz 2024-03-29T11:31:55.649 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38866.log 2024-03-29T11:31:55.649 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.39061.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40275.log 2024-03-29T11:31:55.649 INFO:teuthology.orchestra.run.smithi114.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.38944.log.gz 2024-03-29T11:31:55.649 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.38866.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40080.log 2024-03-29T11:31:55.649 INFO:teuthology.orchestra.run.smithi114.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39061.log.gz 2024-03-29T11:31:55.649 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.40275.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.38866.log.gz 2024-03-29T11:31:55.649 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39412.log 2024-03-29T11:31:55.649 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39334.log 2024-03-29T11:31:55.649 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.40080.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40275.log.gz 2024-03-29T11:31:55.649 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.39412.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41528.log 2024-03-29T11:31:55.649 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.39334.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40080.log.gz 2024-03-29T11:31:55.649 INFO:teuthology.orchestra.run.smithi114.stderr:gzip 86.9% -5 -- replaced with /var/log/ceph/ceph-client.admin.39412.log.gz --verbose 2024-03-29T11:31:55.649 INFO:teuthology.orchestra.run.smithi114.stderr: -- /var/log/ceph/ceph-client.admin.40744.log 2024-03-29T11:31:55.649 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.41528.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39100.log 2024-03-29T11:31:55.650 INFO:teuthology.orchestra.run.smithi114.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39334.log.gz 2024-03-29T11:31:55.650 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.40744.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39178.log 2024-03-29T11:31:55.650 INFO:teuthology.orchestra.run.smithi114.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.41528.log.gz 2024-03-29T11:31:55.650 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.39100.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-29T11:31:55.650 INFO:teuthology.orchestra.run.smithi114.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.40744.log.gz 2024-03-29T11:31:55.650 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.39178.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41606.log 2024-03-29T11:31:55.650 INFO:teuthology.orchestra.run.smithi114.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39100.log.gz 2024-03-29T11:31:55.650 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-osd.3.log: gzip 86.4% -5 -- replaced with /var/log/ceph/ceph-client.admin.39178.log.gz 2024-03-29T11:31:55.650 INFO:teuthology.orchestra.run.smithi114.stderr: --verbose -- /var/log/ceph/ceph-client.admin.38905.log 2024-03-29T11:31:55.650 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40470.log 2024-03-29T11:31:55.650 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.41606.log: /var/log/ceph/ceph-client.admin.38905.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40236.log 2024-03-29T11:31:55.650 INFO:teuthology.orchestra.run.smithi114.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.41606.log.gz 2024-03-29T11:31:55.650 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.40470.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.38905.log.gz 2024-03-29T11:31:55.650 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41488.log 2024-03-29T11:31:55.651 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.40236.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39880.log 2024-03-29T11:31:55.651 INFO:teuthology.orchestra.run.smithi114.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40470.log.gz 2024-03-29T11:31:55.651 INFO:teuthology.orchestra.run.smithi114.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.40236.log.gz 2024-03-29T11:31:55.651 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.41488.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38671.log 2024-03-29T11:31:55.651 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.39880.log: 85.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41645.log 2024-03-29T11:31:55.651 INFO:teuthology.orchestra.run.smithi114.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41488.log.gz 2024-03-29T11:31:55.651 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.38671.log: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.39880.log.gz 2024-03-29T11:31:55.651 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40431.log 2024-03-29T11:31:55.651 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.41645.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39022.log 2024-03-29T11:31:55.651 INFO:teuthology.orchestra.run.smithi114.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.38671.log.gz 2024-03-29T11:31:55.651 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.40431.log: gzip -5 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41645.log.gz 2024-03-29T11:31:55.651 INFO:teuthology.orchestra.run.smithi114.stderr: --verbose -- /var/log/ceph/ceph-client.admin.41449.log 2024-03-29T11:31:55.651 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.39022.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40431.log.gz 2024-03-29T11:31:55.651 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39490.log 2024-03-29T11:31:55.651 INFO:teuthology.orchestra.run.smithi114.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39022.log.gz 2024-03-29T11:31:55.651 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.41449.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39139.log 2024-03-29T11:31:55.652 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.39490.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-29T11:31:55.652 INFO:teuthology.orchestra.run.smithi114.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.41449.log.gz 2024-03-29T11:31:55.652 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.39139.log: gzip 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39490.log.gz 2024-03-29T11:31:55.652 INFO:teuthology.orchestra.run.smithi114.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.38749.log 2024-03-29T11:31:55.652 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41095.log 2024-03-29T11:31:55.652 INFO:teuthology.orchestra.run.smithi114.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39139.log.gz 2024-03-29T11:31:55.652 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.38749.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39802.log 2024-03-29T11:31:55.652 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.41095.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40822.log 2024-03-29T11:31:55.652 INFO:teuthology.orchestra.run.smithi114.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.38749.log.gz 2024-03-29T11:31:55.652 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.39802.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40666.log 2024-03-29T11:31:55.652 INFO:teuthology.orchestra.run.smithi114.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.41095.log.gz 2024-03-29T11:31:55.652 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38513.log 2024-03-29T11:31:55.652 INFO:teuthology.orchestra.run.smithi114.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.39802.log.gz 2024-03-29T11:31:55.652 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.40822.log: /var/log/ceph/ceph-client.admin.40666.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41251.log 2024-03-29T11:31:55.652 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.38513.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39451.log 2024-03-29T11:31:55.653 INFO:teuthology.orchestra.run.smithi114.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.40666.log.gz 2024-03-29T11:31:55.653 INFO:teuthology.orchestra.run.smithi114.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40822.log.gz 2024-03-29T11:31:55.653 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.41251.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.38513.log.gz 2024-03-29T11:31:55.653 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39529.log 2024-03-29T11:31:55.653 INFO:teuthology.orchestra.run.smithi114.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41251.log.gz 2024-03-29T11:31:55.653 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-29T11:31:55.653 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.39451.log: /var/log/ceph/ceph-client.admin.39529.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39924.log 2024-03-29T11:31:55.653 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40978.log 2024-03-29T11:31:55.653 INFO:teuthology.orchestra.run.smithi114.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39529.log.gz 2024-03-29T11:31:55.653 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.39924.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39451.log.gz 2024-03-29T11:31:55.653 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40353.log 2024-03-29T11:31:55.653 INFO:teuthology.orchestra.run.smithi114.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.39924.log.gz 2024-03-29T11:31:55.653 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.40978.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-29T11:31:55.653 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.40353.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40978.log.gz 2024-03-29T11:31:55.653 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40705.log 2024-03-29T11:31:55.654 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40353.log.gz 2024-03-29T11:31:55.654 INFO:teuthology.orchestra.run.smithi114.stderr: --verbose -- /var/log/ceph/ceph-client.admin.34666.log 2024-03-29T11:31:55.654 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.40705.log: /var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-29T11:31:55.654 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.34666.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40705.log.gz 2024-03-29T11:31:55.654 INFO:teuthology.orchestra.run.smithi114.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.34666.log.gz 2024-03-29T11:31:55.654 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41801.log 2024-03-29T11:31:55.654 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41290.log 2024-03-29T11:31:55.654 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.41801.log: 92.8% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-29T11:31:55.654 INFO:teuthology.orchestra.run.smithi114.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41801.log.gz 2024-03-29T11:31:55.654 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40783.log 2024-03-29T11:31:55.654 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.41290.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38827.log 2024-03-29T11:31:55.654 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.40783.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41290.log.gz 2024-03-29T11:31:55.654 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40041.log 2024-03-29T11:31:55.654 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.38827.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40783.log.gz 2024-03-29T11:31:55.654 INFO:teuthology.orchestra.run.smithi114.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.38827.log.gz 2024-03-29T11:31:55.661 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40509.log 2024-03-29T11:31:55.661 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.40041.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35060.log 2024-03-29T11:31:55.661 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.40509.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40041.log.gz 2024-03-29T11:31:55.661 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40627.log 2024-03-29T11:31:55.661 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.35060.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40509.log.gz 2024-03-29T11:31:55.661 INFO:teuthology.orchestra.run.smithi114.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.35060.log.gz 2024-03-29T11:31:55.661 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-29T11:31:55.661 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/ceph-client.admin.40627.log: /var/log/ceph/ceph-mon.c.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40627.log.gz 2024-03-29T11:31:55.665 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-29T11:31:55.665 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34518.log 2024-03-29T11:31:55.666 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-29T11:31:55.666 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-osd.6.log: /var/log/ceph/ceph-client.admin.34518.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34638.log 2024-03-29T11:31:55.666 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-29T11:31:55.666 INFO:teuthology.orchestra.run.smithi176.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34518.log.gz 2024-03-29T11:31:55.666 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-29T11:31:55.666 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.34638.log: 88.3% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-29T11:31:55.666 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-29T11:31:55.666 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34398.log 2024-03-29T11:31:55.666 INFO:teuthology.orchestra.run.smithi176.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34638.log.gz 2024-03-29T11:31:55.666 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34478.log 2024-03-29T11:31:55.666 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.34398.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34678.log 2024-03-29T11:31:55.666 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.34478.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34598.log 2024-03-29T11:31:55.666 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.34678.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34398.log.gz 2024-03-29T11:31:55.666 INFO:teuthology.orchestra.run.smithi176.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34478.log.gz 2024-03-29T11:31:55.667 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-29T11:31:55.667 INFO:teuthology.orchestra.run.smithi176.stderr:gzip 88.0% -- replaced with /var/log/ceph/ceph-client.admin.34678.log.gz 2024-03-29T11:31:55.667 INFO:teuthology.orchestra.run.smithi176.stderr: -5/var/log/ceph/ceph-client.admin.34598.log: --verbose -- /var/log/ceph/ceph.log 2024-03-29T11:31:55.667 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 88.0% -- replaced with /var/log/ceph/ceph-client.admin.34598.log.gz 2024-03-29T11:31:55.667 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T11:31:55.667 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34438.log 2024-03-29T11:31:55.667 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34558.log 2024-03-29T11:31:55.667 INFO:teuthology.orchestra.run.smithi176.stderr: 87.1% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-29T11:31:55.667 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.34438.log: /var/log/ceph/ceph-client.admin.34558.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34438.log.gz 2024-03-29T11:31:55.667 INFO:teuthology.orchestra.run.smithi176.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34558.log.gz 2024-03-29T11:31:55.667 INFO:teuthology.orchestra.run.smithi176.stderr: 93.8% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-29T11:31:55.693 INFO:teuthology.orchestra.run.smithi114.stderr: 92.3% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-29T11:32:07.382 INFO:teuthology.orchestra.run.smithi114.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-29T11:32:07.574 INFO:teuthology.orchestra.run.smithi176.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-29T11:32:07.584 INFO:teuthology.orchestra.run.smithi176.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-29T11:32:07.674 INFO:teuthology.orchestra.run.smithi176.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-29T11:32:11.047 INFO:teuthology.orchestra.run.smithi176.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-29T11:32:11.540 INFO:teuthology.orchestra.run.smithi114.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-29T11:32:11.793 INFO:teuthology.orchestra.run.smithi114.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-29T11:32:13.211 INFO:teuthology.orchestra.run.smithi114.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-29T11:32:46.851 INFO:teuthology.orchestra.run.smithi114.stderr: 92.8% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-29T11:32:47.676 INFO:teuthology.orchestra.run.smithi114.stderr: 92.8% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-29T11:32:47.678 INFO:teuthology.orchestra.run.smithi114.stderr: 2024-03-29T11:32:47.678 INFO:teuthology.orchestra.run.smithi114.stderr:real 0m52.084s 2024-03-29T11:32:47.678 INFO:teuthology.orchestra.run.smithi114.stderr:user 2m38.880s 2024-03-29T11:32:47.678 INFO:teuthology.orchestra.run.smithi114.stderr:sys 0m5.573s 2024-03-29T11:33:05.533 INFO:teuthology.orchestra.run.smithi176.stderr: 92.5% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-29T11:33:05.535 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-29T11:33:05.535 INFO:teuthology.orchestra.run.smithi176.stderr:real 1m9.901s 2024-03-29T11:33:05.535 INFO:teuthology.orchestra.run.smithi176.stderr:user 1m56.570s 2024-03-29T11:33:05.535 INFO:teuthology.orchestra.run.smithi176.stderr:sys 0m4.394s 2024-03-29T11:33:05.535 INFO:tasks.ceph:Archiving logs... 2024-03-29T11:33:05.536 DEBUG:teuthology.misc:Transferring archived files from smithi114:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622690/remote/smithi114/log 2024-03-29T11:33:05.537 DEBUG:teuthology.orchestra.run.smithi114:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-29T11:33:28.102 DEBUG:teuthology.misc:Transferring archived files from smithi176:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622690/remote/smithi176/log 2024-03-29T11:33:28.103 DEBUG:teuthology.orchestra.run.smithi176:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-29T11:33:46.484 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T11:33:46.694 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=7c56ecc12de043eeb07076b5e46955bc Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T11:33:46.698 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-29T11:33:46.706 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-29T11:33:46.735 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-29T11:33:46.735 DEBUG:teuthology.orchestra.run.smithi114:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-29T11:33:46.749 DEBUG:teuthology.orchestra.run.smithi176:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-29T11:33:46.817 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-29T11:33:46.830 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-29T11:33:46.830 DEBUG:teuthology.orchestra.run.smithi114:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:33:46.832 DEBUG:teuthology.orchestra.run.smithi176:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:33:46.846 INFO:teuthology.orchestra.run.smithi114.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:33:46.850 INFO:teuthology.orchestra.run.smithi176.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:33:46.851 INFO:teuthology.orchestra.run.smithi114.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:33:46.851 INFO:teuthology.orchestra.run.smithi114.stdout:=============================================================================== 2024-03-29T11:33:46.851 INFO:teuthology.orchestra.run.smithi114.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 53 +468us[ +468us] +/- 92ms 2024-03-29T11:33:46.852 INFO:teuthology.orchestra.run.smithi114.stdout:^* hv02.front.sepia.ceph.com 3 6 377 56 -115us[ -117us] +/- 55ms 2024-03-29T11:33:46.852 INFO:teuthology.orchestra.run.smithi114.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 54 -275us[ -275us] +/- 87ms 2024-03-29T11:33:46.852 INFO:teuthology.orchestra.run.smithi114.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:33:46.854 INFO:teuthology.orchestra.run.smithi176.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:33:46.854 INFO:teuthology.orchestra.run.smithi176.stdout:=============================================================================== 2024-03-29T11:33:46.854 INFO:teuthology.orchestra.run.smithi176.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 54 +469us[ +469us] +/- 92ms 2024-03-29T11:33:46.854 INFO:teuthology.orchestra.run.smithi176.stdout:^* hv02.front.sepia.ceph.com 3 6 377 56 -113us[ -122us] +/- 55ms 2024-03-29T11:33:46.855 INFO:teuthology.orchestra.run.smithi176.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 55 -287us[ -287us] +/- 87ms 2024-03-29T11:33:46.855 INFO:teuthology.orchestra.run.smithi176.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:33:46.856 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-29T11:33:46.876 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-29T11:33:46.877 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-29T11:33:46.911 DEBUG:teuthology.orchestra.run.smithi114:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-29T11:33:46.980 DEBUG:teuthology.orchestra.run.smithi176:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-29T11:33:47.051 DEBUG:teuthology.orchestra.run.smithi114:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:33:47.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:33:47.082 DEBUG:teuthology.orchestra.run.smithi176:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:33:47.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:33:47.113 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-29T11:33:47.124 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-29T11:33:47.151 INFO:teuthology.task.internal:Duration was 1365.545235 seconds 2024-03-29T11:33:47.152 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-29T11:33:47.170 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-29T11:33:47.171 DEBUG:teuthology.orchestra.run.smithi114:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-29T11:33:47.173 DEBUG:teuthology.orchestra.run.smithi176:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-29T11:33:47.214 INFO:teuthology.orchestra.run.smithi114.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:33:47.218 INFO:teuthology.orchestra.run.smithi176.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:33:47.598 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-29T11:33:47.598 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi114.front.sepia.ceph.com 2024-03-29T11:33:47.598 DEBUG:teuthology.orchestra.run.smithi114:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-29T11:33:47.618 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi176.front.sepia.ceph.com 2024-03-29T11:33:47.618 DEBUG:teuthology.orchestra.run.smithi176:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-29T11:33:47.642 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-29T11:33:47.642 DEBUG:teuthology.orchestra.run.smithi114:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-29T11:33:47.661 DEBUG:teuthology.orchestra.run.smithi176:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-29T11:33:47.712 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-29T11:33:47.713 DEBUG:teuthology.orchestra.run.smithi114:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-29T11:33:47.729 DEBUG:teuthology.orchestra.run.smithi176:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-29T11:33:47.851 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-29T11:33:47.860 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-29T11:33:47.860 DEBUG:teuthology.orchestra.run.smithi114:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-29T11:33:47.887 DEBUG:teuthology.orchestra.run.smithi176:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-29T11:33:47.914 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-29T11:33:47.925 DEBUG:teuthology.orchestra.run.smithi114:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:33:47.930 DEBUG:teuthology.orchestra.run.smithi176:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:33:47.952 INFO:teuthology.orchestra.run.smithi114.stdout:kernel.core_pattern = core 2024-03-29T11:33:47.980 INFO:teuthology.orchestra.run.smithi176.stdout:kernel.core_pattern = core 2024-03-29T11:33:48.011 DEBUG:teuthology.orchestra.run.smithi114:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:33:48.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:33:48.033 DEBUG:teuthology.orchestra.run.smithi176:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:33:48.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:33:48.066 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-29T11:33:48.076 INFO:teuthology.task.internal:Transferring archived files... 2024-03-29T11:33:48.077 DEBUG:teuthology.misc:Transferring archived files from smithi114:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622690/remote/smithi114 2024-03-29T11:33:48.078 DEBUG:teuthology.orchestra.run.smithi114:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-29T11:33:48.114 DEBUG:teuthology.misc:Transferring archived files from smithi176:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622690/remote/smithi176 2024-03-29T11:33:48.115 DEBUG:teuthology.orchestra.run.smithi176:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-29T11:33:48.150 INFO:teuthology.task.internal:Removing archive directory... 2024-03-29T11:33:48.150 DEBUG:teuthology.orchestra.run.smithi114:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-29T11:33:48.152 DEBUG:teuthology.orchestra.run.smithi176:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-29T11:33:48.199 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-29T11:33:48.209 INFO:teuthology.task.internal:Not uploading archives. 2024-03-29T11:33:48.210 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-29T11:33:48.227 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-29T11:33:48.227 DEBUG:teuthology.orchestra.run.smithi114:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-29T11:33:48.230 DEBUG:teuthology.orchestra.run.smithi176:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-29T11:33:48.241 INFO:teuthology.orchestra.run.smithi114.stdout: 265305 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 29 11:33 /home/ubuntu/cephtest 2024-03-29T11:33:48.253 INFO:teuthology.orchestra.run.smithi176.stdout: 266016 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 29 11:33 /home/ubuntu/cephtest 2024-03-29T11:33:48.254 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-29T11:33:48.264 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-29T11:33:48.296 INFO:teuthology.nuke:Checking targets against current locks 2024-03-29T11:33:48.325 DEBUG:teuthology.nuke:shortname: smithi114 2024-03-29T11:33:48.325 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T11:33:48.350 DEBUG:teuthology.nuke:shortname: smithi176 2024-03-29T11:33:48.350 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T11:33:48.354 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi114.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622690', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 11:02:07.795198', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBARmukIBI6GY5wbsB78CZNIyvRkpQev8UQ1peVGN117EtbV8WzHfJBDbP9V0zPW/fCCJVQ6y9x2ZLo7xriBIpQ8='} 2024-03-29T11:33:48.374 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi176.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622690', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 11:02:07.796203', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFy1Kl6VRcmodmVDtmzeaCCslSZmfSmeyp0er6tierAsPgfbqTAsjhhvSyWV3picBPdTnnn9YJHJGAOCuGtBLNU='} 2024-03-29T11:33:48.402 INFO:teuthology.orchestra.console.smithi114:Power off 2024-03-29T11:33:48.403 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-29T11:33:48.426 DEBUG:teuthology.orchestra.console.smithi114:power off output: Chassis Power Control: Down/Off 2024-03-29T11:33:48.426 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:33:48.446 INFO:teuthology.orchestra.console.smithi176:Power off 2024-03-29T11:33:48.447 DEBUG:teuthology.orchestra.console.smithi176:pexpect command: ipmitool -H smithi176.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-29T11:33:48.457 DEBUG:teuthology.orchestra.console.smithi114:check power output: Chassis Power is on 2024-03-29T11:33:48.469 DEBUG:teuthology.orchestra.console.smithi176:power off output: Chassis Power Control: Down/Off 2024-03-29T11:33:48.469 DEBUG:teuthology.orchestra.console.smithi176:pexpect command: ipmitool -H smithi176.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:33:48.495 DEBUG:teuthology.orchestra.console.smithi176:check power output: Chassis Power is on 2024-03-29T11:33:52.458 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:33:52.497 DEBUG:teuthology.orchestra.console.smithi176:pexpect command: ipmitool -H smithi176.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:33:52.571 DEBUG:teuthology.orchestra.console.smithi114:check power output: Chassis Power is on 2024-03-29T11:33:52.609 DEBUG:teuthology.orchestra.console.smithi176:check power output: Chassis Power is on 2024-03-29T11:33:56.574 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:33:56.610 DEBUG:teuthology.orchestra.console.smithi176:pexpect command: ipmitool -H smithi176.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:33:56.686 DEBUG:teuthology.orchestra.console.smithi114:check power output: Chassis Power is off 2024-03-29T11:33:56.721 DEBUG:teuthology.orchestra.console.smithi176:check power output: Chassis Power is off 2024-03-29T11:33:56.787 INFO:teuthology.orchestra.console.smithi114:Power off completed 2024-03-29T11:33:56.822 INFO:teuthology.orchestra.console.smithi176:Power off completed 2024-03-29T11:33:56.927 INFO:teuthology.run:Summary data: description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados tasks/rados_api_tests validater/lockdep} duration: 1365.5452353954315 failure_reason: reached maximum tries (91) after waiting for 540 seconds flavor: default owner: scheduled_amathuri@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=7c56ecc12de043eeb07076b5e46955bc status: fail success: false 2024-03-29T11:33:56.928 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-29T11:33:57.013 INFO:teuthology.run:FAIL